Apparatus and method for providing contents related to augmented reality service between electronic device and wearable electronic device

ABSTRACT

A method and a device capable of supporting various display methods using an electronic device and/or glasses-type wearable electronic device (e.g., AR glasses) in an augmented reality (AR) are provided. An AR providing device for AR services includes a display and a processor. The processor is configured to provide content through an AR screen, detect a specified external object through the AR screen while providing the content, determine a display mode for providing the content, based on detection of the specified external object, control to display the content through a display of the specified external object, based on the determined display mode, and perform control to display the content through a virtual display area associated with the specified external object on the AR screen, based on the determined display mode.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation application, claiming priority under§ 365(c), of an International application No. PCT/KR2022/017539, filedon Nov. 9, 2022, which is based on and claims the benefit of a Koreanpatent application number 10-2021-0153043, filed on Nov. 9, 2021, in theKorean Intellectual Property Office, the disclosure of which isincorporated by reference herein in its entirety.

TECHNICAL FIELD

The disclosure relates to a method and an apparatus capable ofsupporting various display methods using an electronic device and/or aglasses-type wearable electronic device (e.g., AR glasses) in augmentedreality (AR).

BACKGROUND ART

Recently, research and development of extended reality (XR) technologiessuch as virtual reality (VR), augmented reality (AR), and/or mixedreality (MR) have been conducted. VR, AR, and/or MR technologies havebeen used in various fields (e.g., entertainment, infotainment, smarthomes, and/or smart factories), and hardware parts and/or software partsof an electronic device for the same are actively being researched anddeveloped.

For example, a wearable electronic device may superimpose, through anapplication related to an AR service, a variety of digital content(e.g., virtual images) on the real world (or overlay the real world withthe digital content) alone (e.g., a standalone method) or in associationwith at least two devices (e.g., a tethered method), thereby providingone image through a display of the wearable electronic device. Forexample, recently, AR environments such as a tethered AR method in whichan electronic device (e.g., a smartphone) and a wearable electronicdevice are connected to provide virtual content produced by theelectronic device through a display of the wearable electronic deviceand a standalone AR method in which a wearable electronic deviceindependently produces virtual content without connection with anelectronic device and provides the same through a display have beenimplemented.

As described above, in line with the recent technological development ofAR services, the number of users using the AR services is increasing,and thus user needs are also increasing. For example, users who use ARservices have an increasing demand for selecting and displaying contentmore accurately and intuitively in an AR environment.

The above information is presented as background information only toassist with an understanding of the disclosure. No determination hasbeen made, and no assertion is made, as to whether any of the abovemight be applicable as prior art with regard to the disclosure.

DISCLOSURE OF INVENTION Technical Problem

Aspects of the disclosure are to address at least the above-mentionedproblems and/or disadvantages and to provide at least the advantagesdescribed below. Accordingly, an aspect of the disclosure is to providea method and an apparatus capable of supporting screen switching fordisplaying virtual content and content movement between devices in an ARenvironment using an electronic device and/or a wearable electronicdevice.

Another aspect of the disclosure is to provide a method and an apparatuscapable of supporting content operation depending on an AR environmentand a mobile environment by providing content through a display areaand/or a virtual extension area of an electronic device whileimplementing an AR environment using a wearable electronic device.

Another aspect of the disclosure is to provide a method and an apparatuscapable of, in the state in which virtual content is displayed through awearable electronic device, determining a display mode for displayingthe virtual content through interaction between an electronic device andthe wearable electronic device, and providing the virtual contentthrough the electronic device and/or the wearable device, based on thedetermined display mode.

Another aspect of the disclosure is to provide a method and an apparatuscapable of, while providing an AR environment, providing content througha display area of an electronic device or a display area of anelectronic device in the AR environment of a wearable electronic deviceand/or a virtual display area based on an electronic device.

Another aspect of the disclosure is to provide a method and an apparatuscapable of, while providing an AR environment, determining a device toprovide virtual content, based on an operation entity implementing theAR environment and providing virtual content to correspond to a displayarea of the determined device.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments.

Solution to Problem

In accordance with an aspect of the disclosure, an augmented reality(AR) providing device for AR services is provided. The device includes adisplay module and a processor. The processor is configured to providecontent through an AR screen, detect a specified external object throughthe AR screen while providing the content, determine a display mode forproviding the content, based on detection of the specified externalobject, and, based on the determined display mode, perform control todisplay the content through a display of the specified external object,based on the determined display mode, or may perform control to displaythe content through a virtual display area associated with the specifiedexternal object on the AR screen, based on the determined display mode.

In accordance with another aspect of the disclosure, an augmentedreality (AR) providing device for AR services is provided. The deviceincludes a display and a processor. The processor is configured todisplay content through the display, recognize an external devicecorresponding to the external object on the AR screen while displayingthe content through the display, monitor a change in state of theexternal device if the external device is a specified external device,determine a processing entity related to the content, based onidentifying the change in state of the external device, determine adisplay mode of the content, based on the determined processing entity,perform control to display the content on an AR screen through thedisplay, based on the determined display mode, and control the externaldevice to display the content through the display of the externaldevice, based on the determined display mode.

In accordance with another aspect of the disclosure, a method ofproviding an augmented reality (AR) service is provided. The methodincludes providing content through an AR screen, detecting a specifiedexternal object through the AR screen while providing the content,determining a display mode for providing the content, based on thedetection of the specified external object, and, based on the determineddisplay mode, performing control to display the content through adisplay of the specified external object or performing control todisplay the content through a virtual display area associated with thespecified external object on the AR screen.

In order to solve the above problems, various embodiments of thedisclosure may provide a computer-readable recording medium recording aprogram for executing the method in a processor.

Further scope of applicability of the disclosure will become apparentfrom the following detailed description. However, since various changesand modifications within the spirit and scope of the disclosure are tobe clearly understood by those skilled in the art, it should beunderstood that the detailed description and specific embodiments, suchas preferred embodiments of the disclosure, are given by way of exampleonly.

Advantageous Effects of Invention

According to an electronic device and a method of operating the sameaccording to the disclosure, it is possible to more intuitively supportscreen switching for displaying content (e.g., AR content) and contentmovement between devices in an AR environment using an electronic deviceand/or a wearable electronic device. According to the disclosure, it ispossible to provide content through a display area and/or a virtualextension area of an electronic device while implementing an ARenvironment using a wearable electronic device. According to thedisclosure, it is possible to support various content operationsdepending on an AR environment and a mobile environment.

According to the disclosure, it is possible to determine a display modeto display virtual content through interaction between an electronicdevice and a wearable electronic device in the state in which content isdisplayed through the wearable electronic device. According to thedisclosure, it is possible to provide content in various ways through anelectronic device and/or a wearable device, based on the determineddisplay mode. According to the disclosure, it is possible to providecontent in various ways through a display area of an electronic deviceor a display area of an electronic device in an AR environment of awearable electronic device and/or a virtual display area based on anelectronic device while providing the AR environment.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses various embodiments of the disclosure.

BRIEF DESCRIPTION OF DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a block diagram illustrating an example electronic device in anetwork environment according to an embodiment of the disclosure;

FIG. 2 is a diagram illustrating an example of an electronic deviceaccording to an embodiment of the disclosure;

FIG. 3 is a diagram illustrating an example of a wearable electronicdevice according to an embodiment of the disclosure;

FIG. 4 is a diagram illustrating an example of an internal structure ofa wearable electronic device according to an embodiment of thedisclosure;

FIG. 5 is a diagram illustrating an example of the configuration of awearable electronic device according to an embodiment of the disclosure;

FIG. 6 is a diagram illustrating a network environment between anelectronic device and a wearable electronic device according to anembodiment of the disclosure;

FIGS. 7A, 7B, 7C, and 7D are diagrams illustrating an example ofselectively displaying content between an electronic device and awearable electronic device according to various embodiments of thedisclosure;

FIG. 8 is a flowchart illustrating a method of operating an electronicdevice or a wearable electronic device according to an embodiment of thedisclosure;

FIG. 9 is a diagram illustrating an example of a specified interactionfor triggering an operation according to an embodiment of thedisclosure;

FIG. 10 is a flowchart illustrating a method of operating an electronicdevice or a wearable electronic device according to an embodiment of thedisclosure;

FIGS. 11A, 11B, 11C, 11D, 11E, and 11F are diagrams illustrating anexample of operation scenarios of an electronic device or a wearableelectronic device according to various embodiments of the disclosure;

FIGS. 12A, 12B, 12C, 12D, 12E, and 12F are diagrams illustrating anexample of operation scenarios of an electronic device or a wearableelectronic device according to various embodiments of the disclosure;

FIGS. 13A, 13B, and 13C are diagrams illustrating an example ofconfiguring an active window in association with an electronic deviceprovided through an AR screen according to various embodiments of thedisclosure;

FIGS. 14A, 14B, 14C, 14D, 14E, and 14F are diagrams illustrating anexample of operation scenarios of an electronic device or a wearableelectronic device according to various embodiments of the disclosure;

FIGS. 15A, 15B, 15C, 15D, 15E, and 15F are diagrams illustrating anexample of operation scenarios of an electronic device or a wearableelectronic device according to various embodiments of the disclosure;

FIG. 16 is a diagram illustrating an operation method between anelectronic device and a wearable electronic device according to anembodiment of the disclosure;

FIG. 17 is a diagram illustrating an operation method between anelectronic device and a wearable electronic device according to anembodiment of the disclosure;

FIGS. 18A, 18B, and 18C are diagrams illustrating an example ofoperating a display mode between an electronic device and a wearableelectronic device according to various embodiments of the disclosure;

FIGS. 19A and 19B are diagrams illustrating an example of operating adisplay mode between an electronic device and a wearable electronicdevice according to various embodiments of the disclosure;

FIGS. 20A, 20B, and 20C are diagrams illustrating an example ofoperating content mirroring and interaction between an electronic deviceand a wearable electronic device according to various embodiments of thedisclosure; and

FIGS. 21A, 21B, and 21C are diagrams illustrating an example of anoperation of sharing content between an electronic device and a wearableelectronic device according to various embodiments of the disclosure.

Throughout the drawings, like reference numerals will be understood torefer to like parts, components, and structures.

MODE FOR THE INVENTION

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of variousembodiments of the disclosure as defined by the claims and theirequivalents. It includes various specific details to assist in thatunderstanding, but these are to be regarded as merely exemplary.Accordingly, those of ordinary skill in the art will recognize thatvarious changes and modifications of the various embodiments describedherein can be made without departing from the scope and spirit of thedisclosure. In addition, descriptions of well-known functions andconstructions may be omitted for clarity and conciseness.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but are merely used by theinventor to enable a clear and consistent understanding of thedisclosure. Accordingly, it should be apparent to those skilled in theart that the following description of various embodiments of thedisclosure is provided for illustration purpose only and not for thepurpose of limiting the disclosure as defined by the appended claims andtheir equivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces.

FIG. 1 is a block diagram illustrating an example electronic device 101in a network environment 100 according to an embodiment of thedisclosure.

Referring to FIG. 1 , the electronic device 101 in the networkenvironment 100 may communicate with an electronic device 102 via afirst network 198 (e.g., a short-range wireless communication network),or at least one of an electronic device 104 or a server 108 via a secondnetwork 199 (e.g., a long-range wireless communication network).According to an embodiment, the electronic device 101 may communicatewith the electronic device 104 via the server 108. According to anembodiment, the electronic device 101 may include a processor 120,memory 130, an input module 150, a sound output module 155, a displaymodule 160, an audio module 170, a sensor module 176, an interface 177,a connecting terminal 178, a haptic module 179, a camera module 180, apower management module 188, a battery 189, a communication module 190,a subscriber identification module (SIM) 196, or an antenna module 197.In various embodiments, at least one of the components (e.g., theconnecting terminal 178) may be omitted from the electronic device 101,or one or more other components may be added in the electronic device101. In various embodiments, some of the components (e.g., the sensormodule 176, the camera module 180, or the antenna module 197) may beimplemented as a single component (e.g., the display module 160).

The processor 120 may execute, for example, software (e.g., a program140) to control at least one other component (e.g., a hardware orsoftware component) of the electronic device 101 coupled with theprocessor 120, and may perform various data processing or computation.According to an embodiment, as at least part of the data processing orcomputation, the processor 120 may store a command or data received fromanother component (e.g., the sensor module 176 or the communicationmodule 190) in volatile memory 132, process the command or the datastored in the volatile memory 132, and store resulting data innon-volatile memory 134. According to an embodiment, the processor 120may include a main processor 121 (e.g., a central processing unit (CPU)or an application processor (AP)), or an auxiliary processor 123 (e.g.,a graphics processing unit (GPU), a neural processing unit (NPU), animage signal processor (ISP), a sensor hub processor, or a communicationprocessor (CP)) that is operable independently from, or in conjunctionwith, the main processor 121. For example, when the electronic device101 includes the main processor 121 and the auxiliary processor 123, theauxiliary processor 123 may be adapted to consume less power than themain processor 121, or to be specific to a specified function. Theauxiliary processor 123 may be implemented as separate from, or as partof the main processor 121.

The auxiliary processor 123 may control at least some of functions orstates related to at least one component (e.g., the display module 160,the sensor module 176, or the communication module 190) among thecomponents of the electronic device 101, instead of the main processor121 while the main processor 121 is in an inactive (e.g., sleep) state,or together with the main processor 121 while the main processor 121 isin an active state (e.g., executing an application). According to anembodiment, the auxiliary processor 123 (e.g., an image signal processoror a communication processor) may be implemented as part of anothercomponent (e.g., the camera module 180 or the communication module 190)functionally related to the auxiliary processor 123. According to anembodiment, the auxiliary processor 123 (e.g., the neural processingunit) may include a hardware structure specified for artificialintelligence model processing. An artificial intelligence model may begenerated by machine learning. Such learning may be performed, e.g., bythe electronic device 101 where the artificial intelligence is performedor via a separate server (e.g., the server 108). Learning algorithms mayinclude, but are not limited to, e.g., supervised learning, unsupervisedlearning, semi-supervised learning, or reinforcement learning. Theartificial intelligence model may include a plurality of artificialneural network layers. The artificial neural network may be a deepneural network (DNN), a convolutional neural network (CNN), a recurrentneural network (RNN), a restricted boltzmann machine (RBM), a deepbelief network (DBN), a bidirectional recurrent deep neural network(BRDNN), deep Q-network or a combination of two or more thereof but isnot limited thereto. The artificial intelligence model may, additionallyor alternatively, include a software structure other than the hardwarestructure.

The memory 130 may store various data used by at least one component(e.g., the processor 120 or the sensor module 176) of the electronicdevice 101. The various data may include, for example, software (e.g.,the program 140) and input data or output data for a command relatedthereto. The memory 130 may include the volatile memory 132 or thenon-volatile memory 134.

The program 140 may be stored in the memory 130 as software, and mayinclude, for example, an operating system (OS) 142, middleware 144, oran application 146.

The input module 150 may receive a command or data to be used by anothercomponent (e.g., the processor 120) of the electronic device 101, fromthe outside (e.g., a user) of the electronic device 101. The inputmodule 150 may include, for example, a microphone, a mouse, a keyboard,a key (e.g., a button), or a digital pen (e.g., a stylus pen).

The sound output module 155 may output sound signals to the outside ofthe electronic device 101. The sound output module 155 may include, forexample, a speaker or a receiver. The speaker may be used for generalpurposes, such as playing multimedia or playing record. The receiver maybe used for receiving incoming calls. According to an embodiment, thereceiver may be implemented as separate from, or as part of the speaker.

The display module 160 may visually provide information to the outside(e.g., a user) of the electronic device 101. The display module 160 mayinclude, for example, a display, a hologram device, or a projector andcontrol circuitry to control a corresponding one of the display,hologram device, and projector. According to an embodiment, the displaymodule 160 may include a touch sensor adapted to detect a touch, or apressure sensor adapted to measure the intensity of force incurred bythe touch.

The audio module 170 may convert a sound into an electrical signal andvice versa. According to an embodiment, the audio module 170 may obtainthe sound via the input module 150, or output the sound via the soundoutput module 155 or a headphone of an external electronic device (e.g.,an electronic device 102) directly (e.g., wiredly) or wirelessly coupledwith the electronic device 101.

The sensor module 176 may detect an operational state (e.g., power ortemperature) of the electronic device 101 or an environmental state(e.g., a state of a user) external to the electronic device 101, andthen generate an electrical signal or data value corresponding to thedetected state. According to an embodiment, the sensor module 176 mayinclude, for example, a gesture sensor, a gyro sensor, an atmosphericpressure sensor, a magnetic sensor, an acceleration sensor, a gripsensor, a proximity sensor, a color sensor, an infrared (IR) sensor, abiometric sensor, a temperature sensor, a humidity sensor, or anilluminance sensor.

The interface 177 may support one or more specified protocols to be usedfor the electronic device 101 to be coupled with the external electronicdevice (e.g., the electronic device 102) directly (e.g., wiredly) orwirelessly. According to an embodiment, the interface 177 may include,for example, a high definition multimedia interface (HDMI), a universalserial bus (USB) interface, a secure digital (SD) card interface, or anaudio interface.

A connecting terminal 178 may include a connector via which theelectronic device 101 may be physically connected with the externalelectronic device (e.g., the electronic device 102). According to anembodiment, the connecting terminal 178 may include, for example, a HDMIconnector, a USB connector, a SD card connector, or an audio connector(e.g., a headphone connector).

The haptic module 179 may convert an electrical signal into a mechanicalstimulus (e.g., a vibration or a movement) or electrical stimulus whichmay be recognized by a user via his tactile sensation or kinestheticsensation. According to an embodiment, the haptic module 179 mayinclude, for example, a motor, a piezoelectric element, or an electricstimulator.

The camera module 180 may capture a still image or moving images.According to an embodiment, the camera module 180 may include one ormore lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to theelectronic device 101. According to an embodiment, the power managementmodule 188 may be implemented as at least part of, for example, a powermanagement integrated circuit (PMIC).

The battery 189 may supply power to at least one component of theelectronic device 101. According to an embodiment, the battery 189 mayinclude, for example, a primary cell which is not rechargeable, asecondary cell which is rechargeable, or a fuel cell.

The communication module 190 may support establishing a direct (e.g.,wired) communication channel or a wireless communication channel betweenthe electronic device 101 and the external electronic device (e.g., theelectronic device 102, the electronic device 104, or the server 108) andperforming communication via the established communication channel. Thecommunication module 190 may include one or more communicationprocessors that are operable independently from the processor 120 (e.g.,the application processor (AP)) and supports a direct (e.g., wired)communication or a wireless communication. According to an embodiment,the communication module 190 may include a wireless communication module192 (e.g., a cellular communication module, a short-range wirelesscommunication module, or a global navigation satellite system (GNSS)communication module) or a wired communication module 194 (e.g., a localarea network (LAN) communication module or a power line communication(PLC) module). A corresponding one of these communication modules maycommunicate with the external electronic device via the first network198 (e.g., a short-range communication network, such as Bluetooth™wireless-fidelity (Wi-Fi) direct, or infrared data association (IrDA))or the second network 199 (e.g., a long-range communication network,such as a legacy cellular network, a 5^(th) generation (5G) network, anext-generation communication network, the Internet, or a computernetwork (e.g., LAN or wide area network (WAN)). These various types ofcommunication modules may be implemented as a single component (e.g., asingle chip), or may be implemented as multi components (e.g., multichips) separate from each other. The wireless communication module 192may identify and authenticate the electronic device 101 in acommunication network, such as the first network 198 or the secondnetwork 199, using subscriber information (e.g., international mobilesubscriber identity (IMSI)) stored in the subscriber identificationmodule 196.

The wireless communication module 192 may support a 5G network, after a4^(th) generation (4G) network, and next-generation communicationtechnology, e.g., new radio (NR) access technology. The NR accesstechnology may support enhanced mobile broadband (eMBB), massive machinetype communications (mMTC), or ultra-reliable and low-latencycommunications (URLLC). The wireless communication module 192 maysupport a high-frequency band (e.g., the millimeter wave (mmWave) band)to achieve, e.g., a high data transmission rate. The wirelesscommunication module 192 may support various technologies for securingperformance on a high-frequency band, such as, e.g., beamforming,massive multiple-input and multiple-output (massive MIMO), fulldimensional MIMO (FD-MIMO), array antenna, analog beam-forming, orlarge-scale antenna. The wireless communication module 192 may supportvarious requirements specified in the electronic device 101, an externalelectronic device (e.g., the electronic device 104), or a network system(e.g., the second network 199). According to an embodiment, the wirelesscommunication module 192 may support a peak data rate (e.g., 20 gigabitsper second (Gbps) or more) for implementing eMBB, loss coverage (e.g.,164 dB or less) for implementing mMTC, or U-plane latency (e.g., 0.5 msor less for each of downlink (DL) and uplink (UL), or a round trip of 1ms or less) for implementing URLLC.

The antenna module 197 may transmit or receive a signal or power to orfrom the outside (e.g., the external electronic device) of theelectronic device 101. According to an embodiment, the antenna module197 may include an antenna including a radiating element including aconductive material or a conductive pattern formed in or on a substrate(e.g., a printed circuit board (PCB)). According to an embodiment, theantenna module 197 may include a plurality of antennas (e.g., arrayantennas). In such a case, at least one antenna appropriate for acommunication scheme used in the communication network, such as thefirst network 198 or the second network 199, may be selected, forexample, by the communication module 190 (e.g., the wirelesscommunication module 192) from the plurality of antennas. The signal orthe power may then be transmitted or received between the communicationmodule 190 and the external electronic device via the selected at leastone antenna. According to an embodiment, another component (e.g., aradio frequency integrated circuit (RFIC)) other than the radiatingelement may be additionally formed as part of the antenna module 197.

According to various embodiments, the antenna module 197 may form ammWave antenna module. According to an embodiment, the mmWave antennamodule may include a printed circuit board, a RFIC disposed on a firstsurface (e.g., the bottom surface) of the printed circuit board, oradjacent to the first surface and capable of supporting a designatedhigh-frequency band (e.g., the mmWave band), and a plurality of antennas(e.g., array antennas) disposed on a second surface (e.g., the top or aside surface) of the printed circuit board, or adjacent to the secondsurface and capable of transmitting or receiving signals of thedesignated high-frequency band.

At least some of the above-described components may be coupled mutuallyand communicate signals (e.g., commands or data) therebetween via aninter-peripheral communication scheme (e.g., a bus, general purposeinput and output (GPIO), serial peripheral interface (SPI), or mobileindustry processor interface (MIPI)).

According to an embodiment, commands or data may be transmitted orreceived between the electronic device 101 and the external electronicdevice 104 via the server 108 coupled with the second network 199. Eachof the electronic devices 102 or 104 may be a device of a same type as,or a different type, from the electronic device 101. According to anembodiment, all or some of operations to be executed at the electronicdevice 101 may be executed at one or more of the external electronicdevices 102, 104, or 108. For example, if the electronic device 101should perform a function or a service automatically, or in response toa request from a user or another device, the electronic device 101,instead of, or in addition to, executing the function or the service,may request the one or more external electronic devices to perform atleast part of the function or the service. The one or more externalelectronic devices receiving the request may perform the at least partof the function or the service requested, or an additional function oran additional service related to the request, and transfer an outcome ofthe performing to the electronic device 101. The electronic device 101may provide the outcome, with or without further processing of theoutcome, as at least part of a reply to the request. To that end, acloud computing, distributed computing, mobile edge computing (MEC), orclient-server computing technology may be used, for example. Theelectronic device 101 may provide ultra low-latency services using,e.g., distributed computing or mobile edge computing. In an embodiment,the external electronic device 104 may include an internet-of-things(IoT) device. The server 108 may be an intelligent server using machinelearning and/or a neural network. According to an embodiment, theexternal electronic device 104 or the server 108 may be included in thesecond network 199. The electronic device 101 may be applied tointelligent services (e.g., smart home, smart city, smart car, orhealthcare) based on 5G communication technology or IoT-relatedtechnology.

The electronic device according to various embodiments may be one ofvarious types of electronic devices. The electronic devices may include,for example, a portable communication device (e.g., a smartphone), acomputer device, a portable multimedia device, a portable medicaldevice, a camera, a wearable device, a home appliance, or the like.According to an embodiment of the disclosure, the electronic devices arenot limited to those described above.

It should be appreciated that various embodiments of the disclosure andthe terms used therein are not intended to limit the technologicalfeatures set forth herein to particular embodiments and include variouschanges, equivalents, or replacements for a corresponding embodiment.With regard to the description of the drawings, similar referencenumerals may be used to refer to similar or related elements. As usedherein, each of such phrases as “A or B,” “at least one of A and B,” “atleast one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and“at least one of A, B, or C,” may include any one of, or all possiblecombinations of the items enumerated together in a corresponding one ofthe phrases. As used herein, such terms as “1st” and “2nd,” or “first”and “second” may be used to simply distinguish a corresponding componentfrom another, and does not limit the components in other aspect (e.g.,importance or order). It is to be understood that if an element (e.g., afirst element) is referred to, with or without the term “operatively” or“communicatively”, as “coupled with,” “coupled to,” “connected with,” or“connected to” another element (e.g., a second element), the element maybe coupled with the other element directly (e.g., wiredly), wirelessly,or via a third element.

As used in connection with various embodiments of the disclosure, theterm “module” may include a unit implemented in hardware, software, orfirmware, or any combination thereof, and may interchangeably be usedwith other terms, for example, “logic,” “logic block,” “part,” or“circuitry”. A module may be a single integral component, or a minimumunit or part thereof, adapted to perform one or more functions. Forexample, according to an embodiment, the module may be implemented in aform of an application-specific integrated circuit (ASIC).

Various embodiments as set forth herein may be implemented as software(e.g., the program 140) including one or more instructions that arestored in a storage medium (e.g., internal memory 136 or external memory138) that is readable by a machine (e.g., the electronic device 101).For example, a processor (e.g., the processor 120) of the machine (e.g.,the electronic device 101) may invoke at least one of the one or moreinstructions stored in the storage medium, and execute it, with orwithout using one or more other components under the control of theprocessor. This allows the machine to be operated to perform at leastone function according to the at least one instruction invoked. The oneor more instructions may include a code generated by a complier or acode executable by an interpreter. The machine-readable storage mediummay be provided in the form of a non-transitory storage medium. Wherein,the “non-transitory” storage medium is a tangible device, and may notinclude a signal (e.g., an electromagnetic wave), but this term does notdifferentiate between where data is semi-permanently stored in thestorage medium and where the data is temporarily stored in the storagemedium.

According to an embodiment, a method according to various embodiments ofthe disclosure may be included and provided in a computer programproduct. The computer program product may be traded as a product betweena seller and a buyer. The computer program product may be distributed inthe form of a machine-readable storage medium (e.g., compact disc readonly memory (CD-ROM)), or be distributed (e.g., downloaded or uploaded)online via an application store (e.g., PlayStore™), or between two userdevices (e.g., smart phones) directly. If distributed online, at leastpart of the computer program product may be temporarily generated or atleast temporarily stored in the machine-readable storage medium, such asmemory of the manufacturer's server, a server of the application store,or a relay server.

According to various embodiments, each component (e.g., a module or aprogram) of the above-described components may include a single entityor multiple entities, and some of the multiple entities may beseparately disposed in different components. According to variousembodiments, one or more of the above-described components may beomitted, or one or more other components may be added. Alternatively oradditionally, a plurality of components (e.g., modules or programs) maybe integrated into a single component. In such a case, according tovarious embodiments, the integrated component may still perform one ormore functions of each of the plurality of components in the same orsimilar manner as they are performed by a corresponding one of theplurality of components before the integration. According to variousembodiments, operations performed by the module, the program, or anothercomponent may be carried out sequentially, in parallel, repeatedly, orheuristically, or one or more of the operations may be executed in adifferent order or omitted, or one or more other operations may beadded.

FIG. 2 is a diagram illustrating an example of an electronic deviceaccording to an embodiment of the disclosure.

Referring to FIG. 2 , examples of various form factors of an electronicdevice 101 depending on display types are illustrated. The electronicdevice 101 may include various form factors such as a bar type or platetype 210, a foldable type 220, 230, or 240, a rollable type 250, and/ora slidable type 260. The electronic device 101 may be implemented invarious forms, and a display (e.g., the display module 160 in FIG. 1 )may be provided in various ways depending on the implementation form ofthe electronic device 101.

Although it has been illustrated in the disclosure that the electronicdevice 101 has a bar type or plate type appearance, embodiments of thedisclosure is not limited thereto. For example, the illustratedelectronic device 101 may be a part of a foldable electronic device 220,230, or 240, a rollable electronic device 250, or a slidable electronicdevice 260.

The foldable electronic device 220, 230, or 240 may indicate anelectronic device in which two different areas of a display (e.g., thedisplay module 160 in FIG. 1 ) is able to be folded so as tosubstantially face each other or face in opposite directions. Ingeneral, the foldable electronic device 220, 230, or 240, when carried,may be in the state in which two different areas of a display are foldedto face each other or in opposite directions. When the foldableelectronic device 220, 230, or 2530 is used, a user may unfold thedisplay (e.g., the display module 160 in FIG. 1 ) such that twodifferent areas form a substantially flat shape.

The foldable electronic device 220, 230, or 240 may include a formfactor (e.g., 220 or 230) including two display surfaces (e.g., a firstdisplay surface and a second display surface) based on one folding axisand a form factor (e.g., 240) including at least three display faces(e.g., a first display face, a second display face, and a third displayface) based on at least two folding axes. Embodiments of the disclosureare not limited thereto; the number of folding axes provided in theelectronic device 101 is not limited thereto.

The display (e.g., the display module 160 in FIG. 1 ) of the foldableelectronic device 220, 230, or 240 may be folded or unfolded in variousways (e.g., in-folding, out-folding, or in/out-folding) depending on theimplementation form thereof The foldable electronic device 220, 230, or240 may include various foldable types such as a vertical foldable type,a horizontal foldable type, a G foldable type, or a Z foldable type.

The slidable electronic device 260 or the rollable electronic device 250may indicate an electronic device in which a display (e.g., the displaymodule 160 in FIG. 1 ) is deformable to be bent such that at least apart thereof is wound or rolled, or is received inside a housing (notshown). According to a user's need, the slidable electronic device 260or the rollable electronic device 250 may be used with a screen displayarea that is expanded by unfolding (e.g., slide-out) the display (e.g.,the display module 160 in FIG. 1 ) or exposing a larger area of thedisplay to the outside.

The rollable electronic device 250 may include a form factor including aroll-up type display (e.g., a rollable display). The slidable electronicdevice 260 or the rollable electronic device 250 may have the area ofthe display that is exposed to the outside depending on the extent towhich the user unfolds the display (e.g., the display module 160 in FIG.1 ).

FIG. 3 is a diagram illustrating an example of a wearable electronicdevice according to an embodiment of the disclosure.

Referring to FIG. 3 , although a wearable electronic device 301 in theform of glasses (e.g., a glasses-type display device or augmentedreality (AR) glasses) is illustrated in FIG. 3 , the disclosure is notlimited thereto. For example, the wearable electronic device 301 mayinclude various types of devices that include a display and are worn (ormounted) on a part of a user's body (e.g., a face or a head) to provideaugmented reality (AR), mixed reality (MR), and/or virtual reality (VR)services. The wearable electronic device 301 may be implemented in theform of at least one of glasses, goggles, a helmet, or a hat, but is notlimited thereto. The wearable electronic device 301 described below maybe a device including at least some of the elements included in theelectronic device 101 described above with reference to FIG. 1 .Although not mentioned in the following description, the wearableelectronic device 301 according to the disclosure may be construed toinclude various elements described with reference to FIG. 1 .

The wearable electronic device 301 may be worn on a user's face toprovide an image (e.g., an augmented reality image, a mixed realityimage, and/or a virtual reality image) to the user. According to anembodiment, the wearable electronic device 301 may provide an AR servicein which virtual information (or a virtual object) is added to at leasta portion of a real space (or environment). For example, the wearableelectronic device 301 may provide the user with virtual informationoverlaid on the real space corresponding to the field of view (FOV) ofthe wearer.

Referring to FIG. 3 , the wearable electronic device 301 may include aglass member (or window members) 310 disposed at positions correspondingto both eyes (e.g., left and right eyes) of the user, a main frame (orbody part) 340 for fixing the glass member 310, support frames (orsupport members) 350 connected at both ends of the main frame 340 to bemounted on the user's ears, and a camera module 380 (e.g., aphotographing camera).

The glass member 310 may include a first glass 320 corresponding to theuser's left eye and a second glass 330 corresponding to the user's righteye. According to an embodiment, the glass member 310 may be supportedby the main frame 340. For example, the glass member 310 may be fittedinto the opening formed in the main frame 340. The AR image emitted froma display module (e.g., the display module 520 in FIG. 5 ) may beprojected onto the glass member 310.

A waveguide or a transparent waveguide may be formed in at least aportion of the glass member 310. The waveguide may serve to guide the ARimage emitted from the display module to the user's eyes. A detaileddescription of the waveguide according to embodiments of the disclosurewill be made with reference to the description related to the firstglass 320 and the second glass 330 in FIG. 4 .

Although the glass member 310 is implemented such that the first glass320 and the second glass 330 are separated to correspond to the user'sleft and right eyes, respectively, as illustrated in FIG. 3 , accordingto another embodiment of the disclosure, the glass member 310 may beimplemented in the form of one glass, instead of separating the firstglass 320 and the second glass 330.

The main frame 340 and the support frame 350 may be implemented in theform of glasses.

The main frame 340 may have a structure capable of being at leastpartially mounted on the user's nose. The main frame 340 may support theglass member 310. The main frame 340 may be formed of a synthetic resinmaterial. The glass member 310 may be fitted into the opening formed inthe main frame 340, so that the main frame 340 may support the glassmember 310.

The support frame 350 may include a first support frame 360 mounted onthe ear in the first direction (e.g., the left ear) and a second supportframe 370 mounted on the ear in the second direction (e.g., the rightear). For example, the main frame 340 and the support frame 350 (e.g.,the first support frame 360 and the second support frame 370) may becoupled to be folded through a hinge (not shown).

The support frame 350 may be rotatably coupled to the main frame 340.The support frame 350 may include a first support frame 360 and a secondsupport frame 370. The first support frame 360 may be coupled to themain frame 340 on the left side (e.g., a first direction) of the mainframe 340 when viewed from the direction ‘A’. The second support frame370 may be coupled to the main frame 340 on the right side (e.g., asecond direction) of the main frame 340 when viewed from the direction‘A’.

The support frame 350 may be fixed to the main frame 340. For example,the first support frame 360 coupled to the left side of the main frame340 and the second support frame 370 coupled to the right side of themain frame 340 may be coupled to each other. The support frame 350coupled to both sides of the main frame 340 may form a ring shape to beworn to be fitted to the user's head. In addition, the support frame 350may be modified into various shapes in which the wearable electronicdevice 301 is able to be worn on the user's face.

The support frame 350 may be formed to be supported by the user's ear.For example, the wearable electronic device 301 may be worn on theuser's face in such a way that the support frame 350 coupled to the mainframe 340 is supported by the user's ear. The support frame 350 mayrotate relative to the main frame 340. The support frame 350 may rotateto approach the main frame 340, thereby reducing the volume of thewearable electronic device 301.

The display module (e.g., the display module 520 in FIG. 5 ) may outputan AR image produced by the processor 120 of the electronic device 101or a processor (e.g., the processor 590 in FIG. 5 ) of the wearableelectronic device 301. When the display module produces and project anAR image onto the glass member 310, the objects included in the AR imagemay be combined with the visible light L incident from the front (e.g.,the direction in which the user views) through the glass member 310,thereby implementing AR. The display module may be a very smallprojector (e.g., a micro-projector or a pico-projector). For example,the display module may be a laser scanning display (LSD), a digitalmicro-mirror display (DMD), and/or liquid crystal-on-silicon (LCoS).According to an embodiment of the disclosure, the display module may bea transparent display. In this case, the light-emitting device includedin the display module may be directly disposed on the glass member 310.In addition, the display module may be various display devices forimplementing AR.

A pair of glass members 310, a pair of support frames 350, and/or a pairof display modules (e.g., the display module 520 in FIG. 5 ) may beprovided to correspond to the user's left and right eyes. For example,the glass member 310 may include a first glass 320 and a second glass330, and the support frame 350 may include a first support frame 360 anda second support frame 370. According to various embodiments of thedisclosure, some of the above-described elements may be differentbetween a configuration corresponding to the left eye and aconfiguration corresponding to the right eye.

The camera module 380 may include, for example, a photographing camera(e.g., a front photographing camera). For example, the camera module 380may be implemented to be disposed in the main frame 340 so as tophotograph a subject in the front (or the front that the user views) ofthe wearable electronic device 301. The camera module 380 may bedisposed in the central portion (or center point) between the firstglass 320 and the second glass 330 in the main frame 340 so as tophotograph the front of the main frame 340. The front of the main frame340 may indicate a direction in which the user wearing the wearableelectronic device 301 views. The wearable electronic device 301 mayinclude a plurality of other cameras, as well as the camera module 380,and the camera module 380 and the plurality of other cameras will bedescribed in detail with reference to FIG. 4 .

FIG. 4 is a diagram illustrating an example of an internal structure ofa wearable electronic device according to an embodiment of thedisclosure.

Referring to FIG. 4 , a wearable electronic device 301 may be a deviceimplemented in a form worn on the user's face or head. According to anembodiment, the wearable electronic device 301 may include a pluralityof glasses (e.g., the first glass 320 and the second glass 330)respectively corresponding to both eyes (e.g., left and right eyes) ofthe user, and may be implemented in the form of glasses. The wearableelectronic device 301 may provide an image related to an AR service tothe user. According to an embodiment, the wearable electronic device 301may project or display a virtual object onto the first glass 320 and/orthe second glass 330, so that the user may view the real worldrecognized through the first glass 320 and/or the second glass 330 ofthe wearable electronic device 301, on which at least one virtual objectis superimposed.

Referring to FIG. 4 , the wearable electronic device 301 may include amain frame (or body part) 340, a support frame (e.g., a first supportframe 360 and a second support frame 370), and a hinge (e.g., a firsthinge 440-1 and a second hinge 440-2).

The main frame 340 and the support frame (e.g., the first support frame360 and/or the second support frame 370) may have various elements ofthe wearable electronic device 301 mounted thereto. The main frame 340and the support frames 360 and 370 may be operatively coupled throughthe hinges 440-1 and 440-2.

The main frame 340 may include a portion formed to be at least partiallysupported on the user's nose.

The support frames 360 and 370 may include a support member having ashape capable of being supported on the user's ear. The support frames360 and 370 may include a first support frame 360 supported on theuser's left ear and a second support frame 370 supported on the user'sright ear.

The first hinge 440-1 may connect the first support frame 360 and themain frame 340 such that the first support frame 360 is rotatable withrespect to the main frame 340. The second hinge 440-2 may connect thesecond support frame 370 and the main frame 340 such that the secondsupport frame 370 is rotatable with respect to the main frame 340.According to another embodiment of the disclosure, the hinges 440-1 and440-2 of the wearable electronic device 301 may be omitted. For example,the support frames 360 and 370 may be directly coupled to the main frame340 to be fixed.

The main frame 340 may include glasses (e.g., a first glass 320 and asecond glass 330) corresponding to both eyes (e.g., left and right eyes)of the user, display modules (e.g., a first display module 414-1 and asecond display module 414-2), waveguides (e.g., a first waveguide 420and a second waveguide 430), a camera module 380 (e.g., a photographingcamera or a front photographing camera), recognition cameras (e.g., afirst recognition camera 411-1 and a second recognition camera 411-2),eye tracking cameras (e.g., a first eye tracking camera 412-1 and asecond eye tracking camera 412-2), one or more microphones (e.g., afirst microphone 441-1, a second microphone 441-2, and/or a thirdmicrophone 441-3), and one or more lighting members (e.g., a firstlighting member 442-1 and/or a second lighting member 442-2).

The wearable electronic device 301 may display a variety of informationby projecting light produced by the display modules 414-1 and 414-2 ontothe glasses 320 and 330. For example, the light produced by the firstdisplay module 414-1 may be projected onto the first glass 320, and thelight produced by the second display module 414-2 may be projected ontothe second glass 330. At least a portion of the first glass 320 and thesecond glass 330 may be formed of a transparent material (e.g., atransparent member).

The first glass 320 (or a first transparent member) corresponding to theuser's left eye may be coupled to the first display module 414-1, andthe second glass 330 (or a second transparent member) corresponding tothe user's right eye may be coupled to the second display module 414-2.The first glass 320 and the second glass 330 may be formed of a glassplate, a plastic plate, and/or a polymer, and may be made transparent ortranslucent.

The display modules (e.g., the first display module 414-1 and the seconddisplay module 414-2) may include a liquid crystal display (LCD), adigital micro-mirror device (DMD), liquid crystal-on-silicon (LCoS), anorganic light-emitting diode (OLED), or a micro light-emitting diode(micro LED).

The first glass 320 and the second glass 330 may include a condensinglens and/or a waveguide (or a transparent waveguide) (e.g., a firstwaveguide 420 and a second waveguide 430). The waveguides 420, 430 maybe at least partially positioned in a portion of the glasses 320 and330. The first waveguide 420 may be partially positioned on the firstglass 320. The second waveguide 430 may be partially positioned on thesecond glass 330.

The waveguides 420 and 430 may serve to transmit the light produced bythe display modules 414-1 and 414-2 to the user's eyes. Light emittedfrom the display modules 414-1 and 414-2 may be incident on one surface(or one end) of the glasses 320 and 330. The light incident on onesurface of the glasses 320 and 330 may be transmitted to the userthrough waveguides 420 and 430 formed (or positioned) in the glasses 320and 330.

The waveguides 420 and 430 may be made of glass, plastic, or polymer,and may include nanopatterns formed on one surface inside or outside thesame. The nanopattern may include a polygonal or curved gratingstructure. The light incident on one surface of the glasses 320 and 330may be transmitted or reflected by the nano-pattern inside thewaveguides 420 and 430 to be transmitted to the user. The waveguides 420and 430 may include at least one of at least one diffractive element(e.g., a diffractive optical element (DOE) or a holographic opticalelement (HOE)) or a reflective element (e.g., a reflective mirror). Thewaveguides 420 and 430 may guide the light emitted from the displaymodules 414-1 and 414-2 (e.g., light sources) to the user's eyes usingat least one diffractive element or reflective element.

The diffractive element may include an input optical member/outputoptical member (not shown). The input optical member may indicate aninput grating area, and the output optical member (not shown) mayindicate an output grating area. The input grating area may serve as aninput terminal that diffracts (or reflects) the light output from thedisplay modules 414-1 and 414-2 (e.g., micro LEDs) to transmit the sameto the glasses (e.g., the first glass 320 and the second glass 330). Theoutput grating area may serve as an outlet that diffracts (or reflects)the light transmitted to the glasses (e.g., the first glass 320 and thesecond glass 330) of the waveguides 420 and 430 to the user's eyes.

The reflective element may include a total reflection optical element ora total reflection waveguide for total internal reflection (TIR). Totalinternal reflection is one way of guiding light in which an incidentangle is formed such that light (e.g., a virtual image) input throughthe input grating area is reflected about 100% by one surface (e.g., aspecified surface) of the waveguides 420 and 430, thereby transmittingabout 100% of light to the output grating area.

The light emitted from the display modules (e.g., 414-1 and 414-2) maybe guided to the waveguides 420 and 430 through the input opticalmember. The light travelling inside the waveguides 420 and 430 may beguided to the user's eyes through the output optical member.

The display modules 414-1 and 414-2 may include a plurality of panels(or display areas), and the plurality of panels may be positioned on theglasses 320 and 330. At least a portion of the display modules 414-1 and414-2 may be formed of a transparent element. The user may recognize theactual space behind the display modules 414-1 and 414-2 through thedisplay modules 414-1 and 414-2. The display modules 414-1 and 414-2 maydisplay a virtual object (or virtual information) in at least a portionof the transparent element such that it appears to the user that thevirtual object is added to at least a portion of the real space. If thedisplay modules 414-1 and 414-2 are transparent micro LEDs, theconfiguration of the waveguides 420 and 430 may be omitted from theglasses 320 and 330.

The wearable electronic device 301 may include a plurality of cameras(e.g., a first camera, a second camera, and a third camera). Forexample, the first camera (e.g., the camera module 380 in FIG. 3 ) maybe a photographing camera 380 (e.g., an RGB camera) for photographing animage corresponding to the field of view (FoV) of the user and/ormeasuring a distance to an object. The second camera may be eye trackingcamera (eye tracking camera module) 412-1 or 412-2 for identifying thedirection of a user's gaze. The third camera may be a recognition camera(gesture camera module) 411-1 or 411-2 for recognizing a certain space.

The photographing camera 380 may photograph the front direction of thewearable electronic device 301, and the eye tracking cameras 412-1 and412-2 may photograph a direction opposite the photographing direction ofthe photographing camera 380. For example, the first eye tracking camera412-1 may partially photograph the user's left eye, and the second eyetracking camera 412-2 may partially photograph the user's right eye. Thewearable electronic device 301 may display a virtual object (or virtualinformation) related to an AR service, based on image informationrelated to the real space obtained through the photographing camera 380.

According to an embodiment of the disclosure, the wearable electronicdevice 301 may display virtual objects, based on the display moduledisposed to correspond to both eyes of the user (e.g., a first displaymodule 414-1 corresponding to the left eye and/or a second displaymodule 414-2 corresponding to the right eye). According to anotherembodiment of the disclosure, the wearable electronic device 301 maydisplay virtual objects, based on predetermined configurationinformation (e.g., resolution, frame rate, brightness, and/or displayarea).

The photographing camera 380 may include a high-resolution camera suchas a high-resolution (HR) camera and/or a photo video (PV) camera. Forexample, the photographing camera 380 may utilize an autofocus functionand an optical image stabilizer (OIS) function to obtain high-qualityimages. The photographing camera 380 may be implemented as a globalshutter (GS) camera and a rolling shutter (RS) camera in addition to acolor camera.

The eye tracking cameras 412-1 and 412-2 may detect the user's gazedirection (e.g., eye movement). For example, the eye tracking cameras412-1 and 412-2 may detect the user's pupil and track the gazedirection. The tracked gaze direction may be used to move the center ofa virtual image including a virtual object in response to the gazedirection. For example, the eye tracking cameras 412-1 and 412-2 may usea global shutter (GS) camera to detect the pupil and track the rapidpupil movement without dragging of a screen, and the performance andspecifications may be substantially the same between the eye trackingcameras 412-1 and 412-2.

The recognition cameras 411-1 and 411-2 may detect a user gesture withina predetermined distance (e.g., a predetermined space) and/or apredetermined space. The recognition cameras 411-1 and 411-2 may be usedfor 3 DoF or 6 DoF head tracking, hand detection, and/or hand tracking.For example, the recognition cameras 411-1 and 411-2 may be utilized toperform spatial recognition for 6 DoF and a simultaneous localizationand mapping (SLAM) function through depth imaging. The recognitioncameras 411-1 and 411-2 may be utilized for a gesture recognitionfunction for recognizing a user gesture. The recognition cameras 411-1and 411-2 may include cameras including GS. For example, recognitioncameras 411-1 and 411-2 may include a camera including GS, which hasless screen drag (or reduced RS phenomenon) like RS (rolling shutter)cameras, to detect and track quick hand movements and/or fine fingermovements.

The wearable electronic device 301 may detect an eye corresponding to aprimary eye and/or a supplementary eye among the left and/or right eyesof the user using at least one camera 411-1, 411-2, 412-1, 412-2, or380. For example, the wearable electronic device 301 may detect an eyecorresponding to a primary eye and/or a supplementary eye, based on auser's gaze direction for an external object or a virtual object.

One or more cameras (e.g., the photographing camera 380, the eyetracking cameras 412-1 and 412-2, and/or the recognition camera 411-1and 411-2) included in the wearable electronic device 301 illustrated inFIG. 4 may not be limited to specific numbers and positions. Forexample, based on the form (e.g., shape or size) of the wearableelectronic device 301, the number and positions of one or more cameras(e.g., the photographing camera 380, the eye tracking cameras 412-1 and412-2, and/or the recognition camera 411-1 and 411-2) may be variouslychanged.

The wearable electronic device 301 may include one or more lightingmembers (or illumination LEDs) (e.g., the first lighting member 442-1and the second lighting member 442-2) for increasing the accuracy of theone or more cameras (e.g., the photographing camera 380, the eyetracking cameras 412-1 and 412-2, and/or the recognition camera 411-1and 411-2). For example, the first lighting member 442-1 may be disposedin a portion corresponding to the user's left eye, and the secondlighting member 442-2 may be disposed in a portion corresponding to theuser's right eye.

The lighting members 442-1 and 442-2 may be used for different purposesdepending on the position where they are attached to the wearableelectronic device 301. For example, the lighting members 442-1 and 442-2may be used as an auxiliary means to increase accuracy (e.g., ease ofdetecting eye gaze) when photographing the user's pupils with the eyetracking cameras 412-1 and 412-2, and may include an IR LED thatproduces light of an infrared wavelength. The lighting members 442-1 and442-2 may also be used as an auxiliary means of supplementing ambientbrightness if it is not easy to detect a subject to be photographed dueto a dark environment or mixing of various light sources and reflectionlight when photographing a user gesture using the recognition cameras411-1 and 411-2.

The wearable electronic device 301 may include microphones (e.g., afirst microphone 441-1, a second microphone 441-2, and a thirdmicrophone 441-3) for receiving a user's voice and ambient sounds.

The first support frame 360 (or a first housing) and/or the secondsupport frame 370 (a second housing) may include printed circuit boards(PCBs) (e.g., a first printed circuit board 431-1 and a second printedcircuit board 431-2), speakers (e.g., a first speaker 432-1 and a secondspeaker 432-2) for outputting an audio signal, batteries (e.g., a firstbattery 433-1 and a second battery 433-2), and/or hinges (e.g., a firsthinge 440-1 and a second hinge 440-2).

The printed circuit boards 431-1 and 431-2 may include flexible boards(e.g., FPCBs or flexible PCBs). The printed circuit boards 431-1 and431-2 may transmit electrical signals to respective elements (e.g., thecameras, the display modules, the microphones, and/or the speakers) ofthe wearable electronic device 301.

The speakers 432-1 and 432-2 may include a first speaker 432-1 fortransmitting an audio signal to the user's left ear and a second speakerfor transmitting an audio signal to the user's right ear 432-2.

The batteries 433-1 and 433-2 may provide power to the printed circuitboards 431-1 and 431-2 through a power management module (e.g., thepower management module 188 in FIG. 1 ). The wearable electronic device301 may include a plurality of batteries 433-1 and 433-2 through thefirst support frame 360 and the second support frame 370, and supplypower to the printed circuit boards 431-1 and 431-2 through the powermanagement module. For example, the plurality of batteries 433-1 and433-2 may be electrically connected to the power management module.

Although not shown, the wearable electronic device 301 illustrated inFIGS. 3 and 4 may be received in a specified external device (e.g., acase) (not shown). According to an embodiment of the disclosure, thecase may include a function of simply storing and charging the wearableelectronic device 301. According to another embodiment of thedisclosure, the case may include elements such as a camera and/or aprocessor, in addition to charging and storage of the wearableelectronic device 301, and may also be used as an auxiliary arithmeticdevice of the wearable electronic device 301 using the above elements.For example, when received inside the case, the wearable electronicdevice 301 may communicate with the case (e.g., wired communicationand/or wireless communication), and the case may perform some of thefunctions of a host device (e.g., a smartphone).

The wearable electronic device 301 may provide an AR service alone or inassociation with at least one other electronic device (e.g., a hostdevice (e.g., the electronic device 101)). For example, the wearableelectronic device 301 may be connected to a host device (e.g., theelectronic device 101) to operate in a tethered AR method of connectingto a network (e.g., cloud) through the host device, thereby providing ARservices. The wearable electronic device 301 may operate to connect to anetwork (e.g., cloud) in a standalone manner without connection with ahost device (e.g., the electronic device 101), thereby providing ARservices.

FIG. 5 is a diagram illustrating an example of the configuration of awearable electronic device according to an embodiment of the disclosure.

FIG. 5 shows an example of the configuration associated with supportingAR services in the electronic device 301 according to an embodiment. Thewearable electronic device 301 illustrated in FIG. 5 may include theelements of the wearable electronic device 301 described with referenceto FIGS. 3 to 4 .

Referring to FIG. 5 , the wearable electronic device 301 may includeglasses 510, a display module 520, a sensor module 530, a communicationmodule 540, a memory 550, an audio module 560, a camera module 570, abattery 580, and a processor 590. Elements included in the wearableelectronic device 301 may be understood as, for example, hardwaremodules (e.g., circuits).

The glasses 510 (e.g., the first glass 320 and/or the second glass 330in FIGS. 3 and 4 ) may include a condensing lens and/or a transparentwaveguide (e.g., the first waveguide 420 and the second waveguide 430 inFIG. 4 ). For example, the transparent waveguide may be at leastpartially positioned in a portion of the glasses 510. Light emitted fromthe display module 520 may be incident on one end of the glasses 510,and the incident light may be transmitted to the user through thewaveguide formed inside the glasses 510. The waveguide may be made ofglass or polymer and include a nano-pattern, for example, a polygonal orcurved grating structure, formed on one surface inside or outside thesame.

The display module 520 may include a plurality of panels (or displayareas), and the plurality of panels may be positioned on the glasses510. At least a portion of the display module 520 may be formed of atransparent element, and the user may recognize the real space behindthe display module 520 through the display module 520. The displaymodule 520 may display a virtual object (or virtual information) on atleast a portion of the transparent element such that it appears to theuser that the virtual object is added to at least a portion of the realspace.

The sensor module 530 may include a proximity sensor 531, an illuminancesensor 532, and/or a gyro sensor 533.

The proximity sensor 531 may detect an object approaching the wearableelectronic device 301.

The illuminance sensor 532 may measure the brightness around thewearable electronic device 301. The processor 590 may identify thebrightness around the wearable electronic device 301 using theilluminance sensor 532 and change brightness-related configurationinformation of the display module 520, based on the brightness. Forexample, if the surrounding brightness is greater than a predeterminedvalue, the processor 590 may configure the brightness of the displaymodule 520 to be higher to improve the user's visibility.

The gyro sensor 533 may detect the posture and position of the wearableelectronic device 301. For example, the gyro sensor 533 may detectwhether or not the wearable electronic device 301 is properly worn onthe user's head. As another example, the gyro sensor 533 may detectmovement of the wearable electronic device 301 or a user wearing thewearable electronic device 301.

The communication module 540 may correspond to the wirelesscommunication module 192 shown in FIG. 1 . The wearable electronicdevice 301 may perform wireless communication with an external device(e.g., the server 201 and/or other electronic devices 101, 102, and 104in FIG. 1 ) through a network using the communication module 540. Forexample, the wearable electronic device 301 may perform wirelesscommunication with the electronic device 101 and exchange instructionsand/or data with each other.

The wearable electronic device 301 may be at least partially controlledby an external device (e.g., the electronic device 101) through thecommunication module 540. For example, the wearable electronic device301 may perform at least one function under the control of anotherexternal electronic device.

The wearable electronic device 301 may transmit, to an external device(e.g., the server 201 and/or other electronic devices 101, 102, and 104in FIG. 1 ), information on an object (e.g., an image) located in thereal space, obtained through the camera module 570, information on adistance to an object, user gaze information, and/or user gestureinformation through the communication module 540.

The communication module 540 may include an antenna module 545. Forexample, the communication module 540 may support various technologies(e.g., beamforming, multiple-input multiple-output (MIMO), and/or anarray antenna) for securing performance in a specified frequency band.The antenna module 545 may transmit or receive a signal or power to orfrom the outside (e.g., a peripheral device of the wearable electronicdevice 301 and/or a server). According to an embodiment, the antennamodule 545 may include a plurality of antennas (e.g., an array antenna).According to an embodiment, a signal or power may be transmitted orreceived between the communication module 540 and the outside throughthe antenna module 545.

The memory 550 may correspond to the memory 130 described with referenceto FIG. 1 . The memory 550 may store a variety of data used by thewearable electronic device 301 when the wearable electronic device 301provides AR services. The data may include, for example, input data oroutput data for software (e.g., the programs 140 in FIG. 1 ) andcommands related thereto. The memory 550 may store instructions thatcause the processor 590 to operate when executed. The instructions maybe stored as software in the memory 550 and may be executable by theprocessor 590.

The audio module 560 may convert a sound into an electric signal orconvert an electric signal into a sound, based on the control of theprocessor 590.

The camera module 570 may correspond to at least one camera describedwith reference to FIGS. 3 to 4 . The camera module 570 may include aplurality of cameras (e.g., the photographing camera 380, therecognition cameras 411-1 and 411-2, and the eye tracking cameras 412-1and 412-2) as illustrated in FIG. 4 .

The battery 580 may supply power to at least one element of the wearableelectronic device 301. The battery 580 may include the plurality ofbatteries described with reference to FIG. 4 . For example, the battery580 may include a first battery for supplying power in the first supportframe 360 and a second battery for supplying power in the second supportframe 370.

The processor 590 may execute a program (e.g., the programs 140 in FIG.1 ) stored in the memory 550 to control at least one other element(e.g., a hardware or software element), and perform various dataprocesses or computations. The processor 120 may operate to provide anAR service to the user, based on the wearable electronic device 301alone or in connection with the electronic device 101. The processor 120may display, through the display module 520, at least one virtual objectto be viewed such that at least one virtual object (or virtualinformation) is added to the real space corresponding to the gaze (e.g.,FOV or angle of view (AOV)) of the user wearing the wearable electronicdevice 301.

The processor 590 may correspond to the processor 120 described withreference to FIG. 1 . The processor 590 may execute an application(e.g., an AR application) to control a peripheral device connected tothe wearable electronic device 301 through a data communication link onthe AR screen, and perform various data processes and computationsrelated to AR services. As examples of data processes or computations,the processor 590 may store data received through the communicationmodule 540 in the memory 550, process the data stored in the memory 550,store result data in the memory 550, and/or transmit the same to aperipheral device (e.g., the electronic device 101). The processor 590may control the display module 520 to superimpose a variety of digitalcontent (e.g., AR images) on the real world provided through the displaymodule 520, thereby displaying a single image (e.g., an AR screen).

The processor 590 may be included in any one of the first support frame360 and the second support frame 370 of the wearable electronic device301 to operate. The processor 590 may include a first processor (e.g., amain processor) and a second processor (e.g., a coprocessor) in thefirst support frame 360 and the second support frame 370 of the wearableelectronic device 301, respectively, and include a dual system structurethat operates in synchronization with the first processor and the secondprocessor.

FIG. 6 is a diagram illustrating a network environment between anelectronic device and a wearable electronic device according to anembodiment of the disclosure.

Referring to FIG. 6 , an electronic device 101 according to anembodiment of the disclosure may be coupled to a wearable electronicdevice 301. The electronic device 101 and the wearable electronic device301 may be connected (e.g., paired) by wire or wireless communication.For example, the electronic device 101 may be connected with thewearable electronic device 301 through short-range wirelesscommunication 600 such as Bluetooth, Bluetooth Low Energy, Wi-Fi, Wi-FiDirect, or ultra-wide band (UWB).

The electronic device 101 may include a smart phone, a tablet personalcomputer (PC), a notebook, and/or a display device (e.g., a monitor ortelevision (TV)). According to an embodiment, the wearable electronicdevice 301 may include AR glasses, smart glasses, or a head mounteddisplay (HMD).

The wearable electronic device 301 may directly produce data (e.g., anAR image) related to AR services (e.g., produce the same, based onstored or processed data) or obtain the data from the outside (e.g., theelectronic device 101 or a server (e.g., the server 108 in FIG. 1 )),thereby displaying the same through the display module 520. For example,the wearable electronic device 301 may be worn on the user's body (e.g.,face) and superimpose a variety of digital content (e.g., an AR image)on the real world to display a single image (e.g., an AR screen) throughthe display module 520. According to an embodiment, the wearableelectronic device 301 may receive image data (e.g., an AR image) fromthe electronic device 101 and display the received image data togetherwith various objects in the real world through the display module 520.

When connected with the electronic device 101 for communication, thewearable electronic device 301 may periodically transmit information onthe images captured by the camera module 570 of the wearable electronicdevice 301 and the user's gaze information (e.g., FOV and/or AOV) to theelectronic device 101, and/or if there is a change in the state (e.g., achange in position or direction) of the wearable electronic device 301,the wearable electronic device 301 may transmit the same to theelectronic device 101. The wearable electronic device 301, whenconnected with the electronic device 101, may provide (e.g., transmit)information, such as image information, gaze information, deviceinformation, sensing information, function information, and/or locationinformation, to the electronic device 101.

When providing an AR service, the electronic device 101 or the wearableelectronic device 301 may detect (or recognize) a specified interactionrelated to triggering the operation of the disclosure on the AR screenand determine a display mode in which content is to be displayed inresponse to the detection of the specified interaction. The electronicdevice 101 or the wearable electronic device 301, based on thedetermined display mode (e.g., a first display mode or a second displaymode), may operate to display content through the display module 160 ofthe electronic device 101 (e.g., the first display mode) or displaycontent on the AR screen through the display module 520 of the wearableelectronic device 301 (e.g., the second display mode). Differentoperations of the display mode based on detection of a specifiedinteraction according to the disclosure will be described in detail withreference to the drawings to be described later.

FIGS. 7A to 7D are diagrams illustrating an example of selectivelydisplaying content between an electronic device and a wearableelectronic device according to various embodiments of the disclosure.

FIG. 7A shows a state in which a user wearing a wearable electronicdevice 301 views an AR screen provided through a display module 520 ofthe wearable electronic device 301. FIG. 7A shows the state in which theuser views content 710 (or an augmented reality image) (e.g., firstvirtual information 711 and second virtual information 713) superimposedon at least a portion of a real space 700 (or environment) through theAR screen of the wearable electronic device 301. FIG. 7B shows the statein which the user in the state of FIG. 7A moves the electronic device101 into the field of view (FOV) 701 of the user so that the electronicdevice 101 is displayed to overlap the content 710. FIG. 7C shows anexample of displaying the content 710 in a first display mode if aprocessing entity of the content 710 is the electronic device 101. FIG.7D shows an example of displaying the content 710 in a second displaymode if a processing entity of the content 710 is the wearableelectronic device 301The content 710 (or the augmented reality image)may be graphic data corresponding to at least one application, which isprovided as virtual objects to the real space 700 in which the userwearing the wearable electronic device 301 is located.

Referring to FIG. 7A, the wearable electronic device 301 may be in thestate of being worn by the user and displaying various content 710 (orcontent-related virtual information) on at least a portion of the realspace 700 through the display module 520 of the wearable electronicdevice 301. For example, the wearable electronic device 301 maysuperimpose at least one piece of virtual information in the real space700 corresponding to the field of view 701 of the user, therebyproviding the same to the user.

The situation of displaying the content 710 may include the case inwhich the user performs a gesture input and/or eye gaze on a real objectin the real space 700. The situation of displaying the content 710 mayinclude a situation of displaying an execution screen of an application(e.g., a browser, a shopping app, and/or SNS) capable of performing asearch function. The situation of displaying the content 710 may includea situation of displaying an execution screen (e.g., a video playbackscreen or an image playback screen) of an application (e.g., a player)for playing media content. The situation of displaying the content 710may include a situation of executing specified simulation (e.g., asituation of using collaboration tools or a situation of playing games)by the user in a virtual space.

For example, the content 710 may include information provided by acorresponding application, such as virtual information (or virtualobjects or graphic images) corresponding to a first application, virtualinformation corresponding to a second application, and/or virtualinformation corresponding to a third application.

In the state in which the content 710 is provided, the wearableelectronic device 301 may track the user's gaze 701 and provide virtualinformation related to at least one real object in the real space 700,which corresponds to the user's gaze 701 or provide an execution screenof a specified application executed by the user to an AR screen (orvirtual space) through the display module 520. For example, the user mayuse the content 710 corresponding to various real objects in the realspace 700 according to the movement of the user's gaze 701 in the ARservice using the wearable electronic device 301.

Referring to FIG. 7B, the user may place the electronic device 101 inthe real space 700 corresponding to the field of view 701 while usingthe AR service using the wearable electronic device 301. In the state inwhich the content 710 is provided through the wearable electronic device301, the user may place the electronic device 101 in the real space 700visible to the user's field of view to be displayed through the displaymodule 520 of the wearable electronic device 301.

If an interaction (or trigger) for placing the electronic device 101 onthe user's gaze 701 is detected while displaying information on thecontent 710, the electronic device 101 or the wearable electronic device301 may determine a display mode (e.g., a first display mode or a seconddisplay mode) for displaying information on the content 710 depending ona processing entity of the content 710.

The display mode may include a first display mode in which the content710 on the AR screen is displayed through the display module 160 of theelectronic device 101. The display mode may include a second displaymode in which the content 710 on the AR screen is displayed in a virtualspace through the display module 520 of the wearable electronic device301.

The interaction (or trigger) may include a gesture of extending thedisplay module 160 of the electronic device 101 positioned on the user'sgaze 701 through the wearable electronic device 301. The gesture ofextending the display module 160 may include an interaction for slidingout a flexible display and/or a rollable activation interaction.

The processing entity for determining the display mode may differdepending on, for example, a standalone method or a tethered method. Thestandalone method may include a method in which the AR service isprocessed by the wearable electronic device 301. The tethered method mayinclude a method in which the AR service is processed by the electronicdevice 101 and provided by interworking between the electronic device101 and the wearable electronic device 301. For example, in the case ofthe tethered method, the processing entity of the content 710 may be theelectronic device 101 that may operate to display the content 710 in thefirst display mode. As another example, in the case of the standalonemethod, the processing entity of the content 710 may be the wearableelectronic device 301 that may operate to display the content 710 in thesecond display mode.

The display mode determined according to the processing entity may varydepending on whether the profile of the content 710 to be providedcorresponds to a prespecified profile. For example, if the content 710to be provided corresponds to a specified profile, the display mode(e.g., the second display mode) using the virtual space may bedetermined, regardless of whether the processing entity is theelectronic device 101 or the wearable electronic device 301. The profilemay include identification information related to whether or not thecontent 710 is information related to privacy and/or whether the contentis an application category (e.g., phone, personal SNS, and/or finance)specified to be protected by the user. The profile for determining thedisplay mode may be variously pre-configured in the electronic device101 or the wearable electronic device 301 by the user.

Referring to FIG. 7C, FIG. 7C shows an example in which the processingentity is the electronic device 101. For example, FIG. 7C shows anexample of the first display mode in which the content 710 iscontinuously displayed through the display mode 160 of the electronicdevice 101 in the case where the AR service is provided in the tetheredmethod and/or where the content 710 to be provided does not correspondto a specified profile. For example, in the case where the electronicdevice 101 performs processing of the content 710 an applicationcorresponding to the content 710 being executed in relation to ARservices or, the current screen display of the wearable electronicdevice 301 may be stopped, and the content 710 being displayed throughthe AR screen may be displayed through the display module 160 of theelectronic device 101. For example, the electronic device 101 maydisplay the content 710 through the entirety or at least a portion ofthe display area of the display module 160 of the electronic device 101in the first display mode. In an embodiment, the stoppage of the screendisplay is, for example, the state in which the wearable electronicdevice 301 is taken off and may include, for example, the state in whichthe display module 520 or the wearable electronic device 301 is poweredoff.

The first display mode is a mode for displaying the content 710,provided through the AR screen, on the display module 160 of theelectronic device 101 and may be provided in various ways depending on auser interaction according to the form factor of the electronic device101.

As illustrated in example <705A> in FIG. 7C, if the form factor of theelectronic device 101 is a bar type or if the form factor is a rollabletype but there is no rollable activation interaction (e.g., in a closed(or slide-in) state), the electronic device 101 may display the content710 through a specified display area of the display module 160.According to an embodiment, the electronic device 101 may provide theentire area or at least a partial area of the display module 160 as anactive window 720 for displaying the content 710.

As illustrated in example <705B> in FIG. 7C, if the form factor of theelectronic device 101 is a rollable type and if the display module 160is extended by a rollable activation interaction 703 such that anextension area B is further provided in addition to a default area A,the electronic device 101 may display the content 710 through aspecified display area of the display module 160.

The electronic device 101 may maintain the existing display state of thedefault area A of the display module 160 and provide the extension areaB, which is further extended, as an active window 730 for displaying thecontent 710. The degree of extending the extension area B may varydepending on at least one of the number of pieces of content 710, thetype of content 710, the display method of the content 710 (e.g., alandscape mode or a portrait mode), or a display size of the content710.

According to another embodiment of the disclosure, since the userinteraction 703 reflects a clear intention of the user for screenextension, the electronic device 101 may provide the entire area (e.g.,default area A+extension area B) including the default area A and theextension area B of the display module 160 as an active window 740 fordisplaying the content 710.

Referring to FIG. 7D, FIG. 7D shows an example in which the processingentity is the wearable electronic device 301. For example, FIG. 7D showsan example of the second display mode in which the content 710 iscontinuously displayed through the display module 520 of the wearableelectronic device 301 if an AR service is provided in the standalonemethod and/or if an AR service is provided in the tethered method butthe content 710 to be provided corresponds to a specified profile. Forexample, if the wearable electronic device 301 performs processing ofthe content 710 or an application corresponding to the content 710 beingexecuted in relation to AR services, or if the content 710 to beprovided corresponds to a specified profile, the wearable electronicdevice 301 may perform the following. Mapping (e.g., texture mapping orlock) content 710 may be displayed in a default area A of the electronicdevice 101 positioned in the real space 700 displayed in the AR screen,in an extension area B of the electronic device 101, and/or in a virtualarea V produced adjacent to the electronic device 101 through thedisplay module 520 of the wearable electronic device 301. The wearableelectronic device 301 may map and display the content 710 through theentirety or at least a portion of the default area A, the extension areaB, the entire area (e.g., default area A+extension area B), and/or thevirtual area V of the electronic device 101 in the virtual spaceaccording to the second display mode.

The second display mode is a mode for displaying the content 710 througha virtual space provided through the display module 520 of the wearableelectronic device 301 and may be provided in various ways depending onan user interaction on the AR screen according to the form factor of theelectronic device 101. The second display mode may be a mode in whichthe content 710, provided by being mapped to a real object in the realspace 700 through the AR screen, is to be mapped and displayed on thedisplay module 160 of the electronic device 101 on the AR screen.

As illustrated in example <715A> in FIG. 7D, if the form factor of theelectronic device 101 is a bar type or if the form factor is a rollabletype but there is no rollable activation interaction (e.g., in a closed(or slide-in) state), the wearable electronic device 301 may map anddisplay content 710 in the default area A of the display module 160 ofthe electronic device 101, which is recognized from the real space 700provided through the display module 520, and/or in the virtual area Vthat is virtually produced based on the default area A.

The wearable electronic device 301 may map the default area A and/or thevirtual area V of the electronic device 101 recognized from the realspace 700, provided through the display module 520, as active windows760 and 770 for displaying the content 710. The content 710 mapped tothe default area A is provided through the AR screen and may not bedisplayed through the display module 160 of the electronic device 101 inthe real space. In an embodiment, the virtual area V may indicate avirtual display area that is digitally produced corresponding to thedefault area A of the electronic device 101, based on the recognizedelectronic device 101, and is displayed only through the display module520 of the wearable electronic device 301.

For example, the virtual area V may not be an extension area resultingfrom actual extension of the electronic device 101. The user may inputonly a gesture for extension, instead of substantially extending theelectronic device 101, and a virtual extension area may be provided inresponse thereto by replacing the area corresponding to an actuallyextension area by a virtual area V. As described above, the content 710provided through the second display mode may be provided only by the ARscreen provided through the display module 520 of the wearableelectronic device 301, and may be viewed and/or controlled only by theuser wearing the wearable electronic device 301, but may be invisible toother users in the real world.

As illustrated in example <715B> in FIG. 7D, if the form factor of theelectronic device 101 is a rollable type and if the display module 160is extended by a rollable activation interaction 703 such that anextension area B is further provided in addition to a default area A,the wearable electronic device 301 may display the content 710 through aspecified display area of the display module 160 of the electronicdevice 101.

The wearable electronic device 301 may maintain the existing displaystate of the default area A of the electronic device 101 in the realspace 700 provided through the display module 520, configure theextension area B, which is further extended, as an active window 780 fordisplaying the content 710, and may map the content 710 to the activewindow 780, thereby providing the same. According to another embodimentof the disclosure, since the user interaction 703 reflects a clearintention of the user for screen extension, the wearable electronicdevice 301 may configure the entire area (e.g., default area A+extensionarea B) including the default area A and the extension area B of theelectronic device 101 in the real space 700 provided through the displaymodule 520 as an active window 790 for displaying the content 710 andmap the content 710 to the active window 790, thereby providing thesame.

As described above, the content 710 provided through the second displaymode may be provided only by the AR screen provided through the displaymodule 520 of the wearable electronic device 301, and may be viewedand/or controlled only by the user wearing the wearable electronicdevice 301, but may be invisible to other users in the real world.

Although the AR service is provided in the tethered method, in the casewhere the content 710 to be provided corresponds to a specified profile,the electronic device 101 may provide the second display mode by anoperation corresponding to the operation of the wearable electronicdevice 301, which was described with reference to FIG. 7D. Theelectronic device 101 may obtain a variety of information for the seconddisplay mode from the wearable electronic device 301 and, based on theobtained information, transmit data related to an active windowaccording to the second display mode (e.g., coordinate, position, and/orspatial information) and information to be displayed on the activewindow to the wearable electronic device 301 such that the wearableelectronic device 301 provides the second display mode.

An augmented reality (AR) providing device (e.g., the electronic device101 or the wearable electronic device 301) for AR services according toan embodiment of the disclosure may include a display module (e.g., thedisplay module 160 of the electronic device 101 or the display module520 of the wearable electronic device 301) and a processor (e.g., theprocessor 120 of the electronic device 101 or the processor 590 of thewearable electronic device processor 301). The processor 120 or 590 mayoperate to provide content through an AR screen. The processor mayoperate to detect a specified external object through the AR screenwhile providing the content. The processor may operate to determine adisplay mode for providing the content, based on detection of thespecified external object. The processor may operate to control displayof the content through a display module of the specified external object(e.g., the display module 160 of the electronic device 101), based onthe determined display mode. The processor may operate to controldisplay of the content through a virtual display area associated withthe specified external object on the AR screen, based on the determineddisplay mode.

The content may be produced by the AR providing device for AR servicesand provided through the display module of the AR providing device(e.g., the display module 160 of the electronic device 101 or thedisplay module 520 of the wearable electronic device 301), or may beproduced by the AR providing device, transmitted to an external deviceconnected through specified communication, and provided through adisplay module of the external device (e.g., the display module 160 ofthe electronic device 101 or the display module 520 of the wearableelectronic device 301).

The processor 120 or 590 may operate to detect a specified interactionthrough the AR screen, determine whether a target device processing thecontent is an electronic device 101 or a wearable electronic device 301,based on the detection of the specified interaction, if the targetdevice is determined to be the electronic device 101, determine a firstdisplay mode in which the content on the AR screen switches to bedisplayed through the electronic device 101, and if the target device isdetermined to be the wearable electronic device 301, determine a seconddisplay mode in which the content is mapped to the electronic device 101and displayed on the AR screen.

Based on the determination of the first display mode, the processor 120or 590 may operate to perform control to stop displaying a screen of thewearable electronic device 301 and control the display of the contentbeing displayed through the AR screen, based on a display areacorresponding to a display module 160 of the electronic device 101.

The processor 120 or 590 may operate to configure a specified activewindow for the content being displayed on the AR screen through adefault area and/or an extension area of the display module 160 of theelectronic device 101, and control display of the content based on theconfigured active window.

The processor 120 or 590 may operate to control content display based ona virtual display area associated with the electronic device 101 on anAR screen provided through a display module 590 of the wearableelectronic device 301, based on the determination of the second displaymode.

Based on the determination of the second display mode, the processor 120or 590 may operate to map and display the content, based at least on adefault area of the electronic device 101 positioned in a real spacedisplayed on the AR screen through the display module 590 of thewearable electronic device 301, an extension area of the electronicdevice 101, and/or a virtual area produced adjacent to the electronicdevice 101.

Based on the determination of the second display mode, the processor 120or 590 may operate to configure a virtual area corresponding to anextension area of the display module 160 of the electronic device 101 tocorrespond to the extension area of the electronic device 101, configurea specified active window for the content being displayed on the ARscreen through the default area and/or the extension area of the displaymodule 160 of the electronic device 101, and control content displaythrough content mapping based on the configured active window.

If the processor 120 or 590 provides the content according to the seconddisplay mode, the content mapped to the AR screen may be provided onlythrough a display area associated with the electronic device 101 on theAR screen, instead of being displayed through the display module 160 ofthe electronic device 101 in the real space.

The specified interaction may include a first interaction in which theelectronic device 101 enters the real space of the AR screen providedthrough the display module 590 of the wearable electronic device 301, asecond interaction in which a user inputs a specified gesture using theelectronic device 101 after the entry of the electronic device 101,and/or a third interaction in which the user inputs a specified menumanipulation gesture through the electronic device 101 on the AR screenafter the entry of the electronic device.

The processor 120 or 590 may operate to determine whether or not thecontent corresponds to a specified profile when determining the targetdevice processing the content and, if the content corresponds to thespecified profile, determine the second display mode so as to providecontent of the specified profile through the AR screen, regardless ofthe target device processing the content.

An augmented reality (AR) providing device (e.g., the electronic device101 or the wearable electronic device 301) for AR services according toan embodiment of the disclosure may include a display module (e.g., thedisplay module 160 of the electronic device 101 or the display module520 of the wearable electronic device 301) and a processor (e.g., theprocessor 120 of the electronic device 101 or the processor 590 of thewearable electronic device processor 301). The processor 120 or 590 mayoperate to display content through the display module. The processor mayoperate to recognize an external device corresponding to the externalobject from the AR screen while displaying the content through thedisplay module. The processor may operate to monitor a change in stateof the external device if the external device is a specified externaldevice. The processor may operate to determine a processing entityrelated to the content, based on identifying a change in the state ofthe external device. The processor may operate to determine the displaymode of the content, based on the determined processing entity. Theprocessor may operate to display the content on the AR screen through adisplay module, based on the determined display mode. The processor mayoperate to control the external device to display the content throughthe display module of the external device, based on the determineddisplay mode.

Operation methods of the electronic device 101 and the wearableelectronic device 301 according to various embodiments of the disclosureare described below.

Operations performed by the electronic device 101 may be executed by theprocessor 120 of the electronic device 101 including various processingcircuits and/or executable program elements. The operations performed bythe electronic device 101 may be stored in the memory 130 and executedby instructions that, when executed, cause the processor 120 to operate.

The operations performed by the wearable electronic device 301 describedbelow may be executed by the processor 590 of the wearable electronicdevice 301 including various processing circuits and/or executableprogram elements. Operations performed by the wearable electronic device301 may be stored in the memory 550 and executed by instructions that,when executed, cause the processor 590 to operate.

The operations of the electronic device 101 and the wearable electronicdevice 301 may be distinguished depending on a processing entityaccording to whether the AR service is a standalone method or a tetheredmethod, and the operation of displaying content by operating the displaymode of the disclosure may be similar or the same therebetween,excluding the operation of performing data communication as necessary.For example, the electronic device 101 and the wearable electronicdevice 301 may perform a series of operations of rapidly switchingbetween screens in the situation of using AR, enabling content movementbetween the electronic device 101 and the wearable electronic device301, configuring an active window, based on a processing entity and/or aspecified profile, and displaying content through a corresponding activewindow.

FIG. 8 is a flowchart illustrating a method of operating an electronicdevice or a wearable electronic device according to an embodiment of thedisclosure.

FIG. 9 is a diagram illustrating an example of a specified interactionfor triggering the operation according to an embodiment of thedisclosure.

FIG. 8 shows an example of an operation of providing content through anelectronic device 101 or a wearable electronic device 301 depending on adisplay mode (e.g., a first display mode or a second display mode) fordisplaying content, based on a specified interaction, in a situation inwhich the electronic device 101 or the wearable electronic device 301provides an AR service according to a tethered method or a standalonemethod. In an embodiment, the operation illustrated in FIG. 8 may beperformed by the electronic device 101 or the wearable electronic device301, and, in the following description of FIG. 8 , the electronic device101 or the wearable electronic device 301 will be referred to as an ARproviding device for convenience of description. The electronic device101 and the wearable electronic device 301 may further include anoperation of receiving, processing, and transmitting related data for anAR service depending on the entity that performs the operation, and adescription thereof will be made with reference to the drawings to bedescribed later.

Referring to FIG. 8 , in operation 801, an AR providing device (e.g.,the electronic device 101 or the wearable electronic device 301) or aprocessor of the AR providing device (e.g., the processor 120 of theelectronic device 101 or the processor 590 of the wearable electronicdevice 301) may provide content through a display module 520 of thewearable electronic device 301. The AR providing device, the wearableelectronic device 301 may be in the state of being worn on the user anddisplaying at least one piece of content (or virtual information)superimposed on at least a portion of a real space on an AR screenthrough the display module 520 of the wearable electronic device 301.

For example, the wearable electronic device 301 may superimpose at leastone piece of content in the real space corresponding to the field ofview of the user, thereby providing the content to the user. Thewearable electronic device 301 may display, through the display module520 of the wearable electronic device 301, content that is directlyproduced for the AR service by the wearable electronic device 301 (e.g.,produced based on stored or processed data) or content that is producedby the electronic device 101 and obtained from the electronic device 101through specified communication. According to an embodiment, the usermay browse AR content provided through the display module 520 of thewearable electronic device 301.

In operation 803, the AR providing device may detect a specifiedinteraction through the AR screen. In an embodiment, an example of aspecified interaction is shown in FIG. 9 .

Referring to FIG. 9 , the specified interaction may include a firstinteraction (example 901) of recognizing the electronic device 101entering the real space of the AR screen provided through the displaymodule 520 of the wearable electronic device 301, a second interaction(example 903) of recognizing a specified gesture (e.g., a slide-outgesture or a rollable extension gesture) input by a user using theelectronic device 101 after the entry of the electronic device 101, or athird interaction (example 905) of recognizing a specified menumanipulation gesture (e.g., a touch input) input by the user through theelectronic device 101 on the AR screen after the entry of the electronicdevice 101.

The second interaction may include a gesture of substantially extendingthe extension area of the electronic device 101. In another embodiment,the second interaction may not be a substantial extension of theextension area of the electronic device 101. For example, the user mayinput only an extending gesture without substantially extending theelectronic device 101, and the AR providing device may recognize theuser's gesture to identify the second interaction. The AR providingdevice may detect a user's gesture according to the second interactionand/or the third interaction, based on the user hand detecting and/orhand tracking on the AR screen.

The example of the specified interaction is intended only to helpunderstanding of the disclosure and is not intended to limit theembodiment of the disclosure. For example, the second interaction in thespecified interaction may include a real extension or virtual extensiontrigger for the electronic device 101 and include, for example, an inputto a specified hardware button of the electronic device 101 forextension of the extension area (e.g., a click input or a long-pressinput) and an input to a specified software button displayed through thedisplay module 160 of the electronic device 101 for extension of theextension area (e.g., a touch input or a long-press input). The secondinteraction may include various specified inputs as an extension triggercapable of substantially or virtually extending the extension area ofthe electronic device 101.

Referring back to FIG. 8 , in operation 805, the AR providing device maydetermine an operation entity for processing the content. The operationentity for processing may be determined differently depending on amethod of providing an AR service. For example, in the tethered method,content may be produced by the electronic device 101 and provided to thewearable electronic device 301, and the processing entity of the contentmay be the electronic device 101. As another example, in the case of thestandalone method, content may be directly produced and provided by thewearable electronic device 301, and the processing entity of the contentmay be the wearable electronic device 301.

For example, in the tethered method, the electronic device 101 has thecontrol right of the overall operation for the AR service, the ARservice is controlled by the electronic device 101, and the wearableelectronic device 301 may obtain content from the electronic device 101,thereby providing the content to the user. As another example, in thestandalone method, the wearable electronic device 301 has the controlright of the overall operation for the AR service, and the AR service iscontrolled by the wearable electronic device 301, and the contentdirectly produced by the wearable electronic device 301 may be provided.The AR providing device may further determine whether or not the contentto be provided corresponds to a specified profile, in addition to theoperation entity for processing. For example, the AR providing devicemay operate to preferentially provide content of a specified profilethrough the AR screen, regardless of the processing entity. An operationusing the specified profile will be described later.

In operation 807, the AR providing device, based on the result ofdetermining the operation entity, may determine whether a target devicefor displaying content is the electronic device 101 or the wearableelectronic device 301.

If it is determined that the target device is the electronic device 101in operation 807 (e.g., “Yes” in operation 807), the AR providing devicemay determine a first display mode in operation 809. If the operationentity is the electronic device 101, the AR providing device maydetermine a first display mode in which the content on the AR screenswitches to be displayed through the electronic device 101.

In operation 811, based on the determination of the first display mode,the AR providing device may perform processing to display content, basedon a display area corresponding to the display module 160 of theelectronic device 101. If the electronic device 101 performs processingof content or an application corresponding to the content being executedin relation to AR services, the AR providing device may operate to stopdisplaying the current screen of the wearable electronic device 301(e.g., turn off the display module 520 or the wearable electronic device301 in the state in which the wearable electronic device 301 is takenoff) and display the content being displayed through the AR screenthrough the display module 160 of the electronic device 101.

If the processing entity related to the content is the electronic device101, the AR providing device may operate to stop displaying the ARscreen of the wearable electronic device 301, configure a specifiedactive window (e.g., a window based on a default area, a window based onan extension area, or a window based on the entire area) for the contentbeing displayed on the AR screen through a default area and/or anextension area of the display module 160 of the electronic device 101,and display the content, based on the configured active window.

If the processing entity is the electronic device 101, the AR providingdevice may configure an active window to correspond to a form factor(e.g., a bar type or rollable type) of the electronic device 101. Forexample, in the case of the bar-type electronic device 101, the ARproviding device may configure an active window based on the defaultarea. As another example, in the case of the rollable type electronicdevice 101, the AR providing device may configure an active window basedon at least one of the default area, a partially extension area, theentire extension area, or the entire area.

If it is determined that the target device is the wearable electronicdevice 301 in operation 807 (e.g., “No” in operation 807), the ARproviding device may determine a second display mode in operation 813.For example, if the operation entity is the wearable electronic device301, the AR providing device may determine a second display mode inwhich the content on the AR screen is mapped to the actual electronicdevice 101 on the AR screen and displayed.

In operation 815, based on the determination of the second display mode,the AR providing device, may perform processing to display the content,based on a virtual display area associated with the electronic device101, on the AR screen provided through the display module 520 of thewearable electronic device 301. If the wearable electronic device 301performs processing of the content or an application corresponding tothe content being executed in relation to AR services, or if the contentto be provided corresponds to a specified profile, the AR providingdevice may process content display, based on the determination of thesecond display mode. The AR providing device may operate to map (e.g.,texture mapping) and display the content, based at least on a defaultarea of the electronic device 101 positioned in the real space displayedon the AR screen through the display module 520 of the wearableelectronic device 301, an extension area of the electronic device 101,and/or a virtual area produced adjacent to the electronic device 101.For example, the AR providing device may display the content through theentirety or at least a portion of the default area of the electronicdevice 101, the extension area, the entire area (e.g., default area+extension area), and/or the virtual area in the virtual space accordingto the second display mode.

If the processing entity related to content is the wearable electronicdevice 301, the AR providing device may operate to configure a virtualarea corresponding to the extension area of the display module 160 ofthe electronic device 101 to correspond to the extension area of theelectronic device 101, configure a specified active window (e.g., awindow based on the default area or a window based on the entire screen)for the content being displayed on the AR screen through the defaultarea and/or the extension area of the display module 160 of theelectronic device 101, and map and display the content, based on theconfigured active window.

As described above, the AR providing device (e.g., the electronic device101 or the wearable electronic device 301) according to the disclosuremay include a processor (e.g., the processor 120 of the electronicdevice 101 or the processor 590 of the wearable electronic device 301).The processor may operate to provide first content (and/or informationrelated to the content) through an AR screen of the wearable electronicdevice 301. The processor may operate to detect a specified interaction(e.g., a specified external object (e.g., detect the electronic device101 entering an AR screen and/or detect a specified gesture)) throughthe AR screen while providing the content through the AR screen.

Based on the detection of the specified interaction, the processor maydetermine a display mode for second content (e.g., the first content orother content obtained by processing the first content), based on aprocessing entity providing the second content.

Based on the determined display mode (e.g., a first display mode or asecond display mode), the processor may operate to display the secondcontent through the display module 160 of the electronic device 101 ormap and display the second content to and on a display area associatedwith a specified external object (e.g., the actual electronic device 101on the AR screen) on the AR screen through the display module 520 of thewearable electronic device 301.

The second content mapped to the AR screen may be provided only throughthe display area associated with the electronic device 101 on the ARscreen, instead of being displayed through the display module 160 of theelectronic device 101 in the real world.

If the processing entity related to the content is the wearableelectronic device 301, the processor may operate to monitor a change instate of the electronic device 101 (e.g., interaction for rollableactivation) in the AR screen while displaying the content on the ARscreen of the wearable electronic device 301, based on identification ofthe change in state of the electronic device 101, determine a displaymode of the content, based on the processing entity related to thecontent, and display the content on an AR screen through the displaymodule 520 of the wearable electronic device 301 or display the contentthrough the display module 160 of the electronic device 101, based onthe determined display mode.

FIG. 10 is a flowchart illustrating a method of operating an electronicdevice or a wearable electronic device according to an embodiment of thedisclosure.

FIGS. 11A to 11F are diagrams illustrating an example of operationscenarios of an electronic device or a wearable electronic deviceaccording to various embodiments of the disclosure.

Referring to FIGS. 10 and 11A to 11F, the operations illustrated thesefigures may be performed by the electronic device 101 or the wearableelectronic device 301 depending on the tethered method or the standalonemethod for providing AR services. Hereinafter, in the description ofFIG. 10 , the electronic device 101 or the wearable electronic device301 will be referred to as an AR providing device for convenience ofdescription. The electronic device 101 and the wearable electronicdevice 301 may further include an operation of receiving, processing,and transmitting related data for AR services depending on the entitythat performs the operation, and a description thereof will be made withreference to the drawings to be described later.

Referring to FIG. 10 , in operation 1001, an AR providing device (e.g.,the electronic device 101 or the wearable electronic device 301) or aprocessor of the AR providing device (e.g., the processor 120 of theelectronic device 101 or the processor 590 of the wearable electronicdevice 301) may provide content through a display module 520 of thewearable electronic device 301. An example of this is shown in FIG. 11A.Referring to FIG. 11A, the wearable electronic device 301 may be worn onthe user, and the AR providing device may display content (or virtualinformation) 1110 superimposed on at least a portion of a real space onan AR screen 1100 through the display module 520 of the wearableelectronic device 301. For example, the AR providing device maysuperimpose content 1110 (e.g., first content 1101, second content 1102,and third content 1103) in the real space corresponding to the field ofview of the user, thereby providing the same to the user.

In operation 1003, the AR providing device may determine whether or notthe electronic device 101 is detected in the AR screen. The AR providingdevice may track whether or not the electronic device 101 enters thereal space provided through the display module 520 of the wearableelectronic device 301. An example of this is shown in FIG. 11A.Referring to FIG. 11A, the user may move the electronic device 101 intothe AR screen 1100 provided through the display module 520 of thewearable electronic device 301. For example, the user may move theelectronic device 101 to a corresponding position to be displayedthrough the AR screen 1100, and the AR providing device may identify theelectronic device 101 moving into the AR screen 1100, based on trackingof an object and/or gaze on the AR screen 1100.

If the electronic device 101 is not detected in operation 1003 (e.g.,“No” in operation 1003), the AR providing device may return to operation1001 and perform operation 1001 and subsequent operations.

If the electronic device 101 is detected in operation 1003 (e.g., “Yes”in operation 1003), the AR providing device may provide an affordance(or guide) for a virtual area in operation 1005. If the AR providingdevice identifies the electronic device 101 on the AR screen and detectsa specified interaction (e.g., a rollable activation interaction) forthe electronic device 101, the AR providing device may provide anaffordance (or guide) for a virtual area (e.g., the area expected byrollable extension) in response to the specified interaction. An exampleof this is shown in FIG. 11B. Referring to FIG. 11B, the AR providingdevice may provide an affordance 1120, based on the electronic device101 or the display module 160 of the electronic device 101. The ARproviding device may provide a virtual window guide corresponding to adefault area and/or a substantial extension area of the electronicdevice 101 as the affordance 1120. The provision of the affordance 1120will be described with reference to the drawings below.

In operation 1007, the AR providing device may track whether or not theaffordance overlaps content. The user may move the electronic device 101such that the affordance at least partially overlaps content. The ARproviding device may track the electronic device 101 in the AR screen(e.g., track an object) and move the affordance together in response tothe movement of the electronic device 101. An example of this is shownin FIG. 11C. Referring to FIG. 11C, the affordance 1120 and at least aportion of the content 1110 (e.g., the first content 1101) may overlapeach other on the AR screen 1100.

If the affordance 1120 and the content 1110 (e.g., the first content1101) overlap, the AR providing device may activate the overlappingcontent 1110 (e.g., the first content 1101) (e.g., move and display thecontent forward of the affordance 1120), thereby distinguishing theoverlapping content 1110 (e.g., the first content 1101) from thenon-overlapping content 1110 (e.g., the second content 1102 and thethird content 1103). For example, the AR providing device may highlightand provide the overlapping content 1110 (e.g., the first content 1101)to be associated with the affordance 1120 so that the user mayintuitively recognize the overlapping content 1110.

The electronic device 101 may be a bar-type electronic device in whichthe display module 160 is not extended and has a fixed size, or may be arollable type electronic device in which the display module 160 isextendable. For example, the electronic device 101 may include asmartphone, a tablet PC, a notebook computer, and/or a display device(e.g., a monitor or TV) including a display module 160 having a fixedsize or a variable size.

If the electronic device 101 is a bar-type electronic device, anaffordance may be provided based on a display area (e.g., a defaultarea) of the display module 160 of the electronic device 101, and theaffordance and content may be mapped in the display area. If theelectronic device 101 is a rollable type electronic device and thedisplay module 160 of the electronic device 101 is in an extended state,an affordance may be provided based on a display area (e.g., defaultarea +extension area) of the display module 160, and the affordance andcontent may be mapped in the display area. If the electronic device 101is a rollable type electronic device and if the display module 160 ofthe electronic device 101 is in a non-extended state, the display module160 may provide an affordance, based on an extendable virtual area(e.g., a virtual area having a size corresponding to the extensionarea), and the affordance and content may be mapped in the virtual area.

The content in the AR screen provided through the display module 520 ofthe wearable electronic device 301 may include space-oriented contentthat is mapped to the real space and the position of which is adaptivelymovable according to the user's gaze or the user's movement (e.g., headmovement) (e.g., the content is moved to always be visible within theuser's gaze, that is, fixedly displayed in a specified area in thedisplay module 520). As another example, the content in the AR screenprovided through the display module 520 of the wearable electronicdevice 301 may include device-oriented (or object-oriented) content thatis not mapped to the real space but fixedly mapped to a specific space(or an area where a corresponding object is located in the real space)on the AR screen of the display module 520 so as not to be movableaccording to the user's gaze or the user's movement.

An operation of mapping (e.g., overlapping) the affordance and thecontent may vary depending on, for example, whether the content isspace-oriented content or device-oriented content. In the case ofspace-oriented content, the user may overlap content and the affordanceby directly moving the electronic device 101 within the AR screen of thedisplay module 520 of the wearable electronic device 301. In the case ofdevice-oriented content, the user may move the user's gaze (e.g., movethe head) to the display module 160 (e.g., the default area, theextension area, the entire area, or the virtual area) of the electronicdevice 101 within the AR screen of the display module 520 of thewearable electronic device 301, and the content may be moved on the ARscreen in response to the movement of the user's gaze so that thecontent may overlap the affordance in the corresponding area of theelectronic device 101. The movement of content on the AR screen is notlimited to the user's gaze and may be performed, for example, by a usergesture on the AR screen (e.g., a user's gesture of selecting and movingcontent on the AR screen).

In operation 1009, the AR providing device may determine whether or notan overlap between the affordance and the content is detected. The ARproviding device may determine whether or not the content overlaps theposition of the affordance corresponding to the movement of theelectronic device 101. The AR providing device may perform operations1007 to 1009 for a specified period of time.

If no overlap between the affordance and the content is detected inoperation 1009 (e.g., “No” in operation 1009), the AR providing devicemay perform a user-specified operation in operation 1011. If no overlapbetween the affordance and the content is detected for a specifiedperiod of time, the AR providing device may remove the affordance fromthe AR screen and continue to provide the content. The AR providingdevice may detect a specified input of the user on the AR screen, removethe affordance in response to the specified input, and perform anoperation (or function) (e.g., change the content) corresponding to thespecified input.

If an overlap between the affordance and the content is detected inoperation 1009 (e.g., “Yes” in operation 1009), the AR providing devicemay detect a specified interaction (or an operation trigger) inoperation 1013. If an overlap between the affordance and the content isdetected, the AR providing device may determine an interaction oftriggering an operation (e.g., an operation trigger) for displaying thecorresponding content selected by the overlap with the affordancethrough a specified target device. An example of this is shown in FIG.11D.

Referring to FIG. 11D, the user may input a specified interaction 1130for selecting and determining the first content 1101, based on anoverlap between the affordance 1120 and the first content 1101 on the ARscreen 1100, and displaying the first content 1101 through a targetdevice. The specified interaction may include, for example, a slide-outor rollable extension gesture, an input to a specified hardware buttonfor an extension trigger, or an input to a specified software button foran extension trigger.

In operation 1015, the AR providing device may determine a target devicefor displaying content, based on detection of the specified interaction.The AR providing device may determine a display mode (e.g., a firstdisplay mode or a second display mode) corresponding to a target devicefor displaying the content selected by the overlap. For example, the ARproviding device may determine a display mode and a target devicecorresponding thereto, based on a content processing entity and/or aspecified profile of the content.

In operation 1017, the AR providing device may produce an active windowfor the content, based on the target device. According to an embodimentof the disclosure, if the target device is the electronic device 101,the AR providing device may configure an active window for the content,based on the display area of the electronic device 101. According toanother embodiment of the disclosure, if the target device is thewearable electronic device 301, the AR providing device may configure anactive window for the content, based on the display area of theelectronic device 101 displayed on the AR screen of the wearableelectronic device 301 and/or the virtual area adjacent to the displayarea.

When configuring an active window, the AR providing device may configurethe active window in at least one of the default area, the extensionarea, or the entire area, based on the overlapping state (or overlappingarea) of the affordance and the content according to the user'sintention. For example, if the content (e.g., the first content 1101)overlaps the affordance, the AR providing device may configure an activewindow to display the content in the extension area. As another example,if the content (e.g., the first content 1101) overlaps both theelectronic device 101 (e.g., the default area of the electronic device101) and the affordance on the AR screen, the AR providing device mayconfigure an active window to display the corresponding content in theentire area (e.g., default area +extension area (or virtual area)). Asanother example, if one piece of content (e.g., the first content 1101)is disposed in the affordance, the AR providing device may configure anactive window to display the corresponding content in the entire area(e.g., default area+extension area).

As another example, if a plurality of content (e.g., the first content1101 and the second content 1102) overlaps the affordance, the ARproviding device may configure an active window in each of the defaultarea and the extension area to display the plurality of pieces ofcontent in the default area and the extension area, respectively. Asanother example, one piece of content (e.g., the first content 1101)among a plurality of pieces of content overlaps the affordance and ifanother piece of content (e.g., the second content 1102) thereofoverlaps the electronic device 101 (e.g., the default area of theelectronic device 101), for example, if each piece of content isdisposed according to the user's intention, the AR providing device mayconfigure an active window for the first content 1101 in the extensionarea and an active window for the second content 1102 in the defaultarea so as to display the content to conform to the dispositionaccording to the user's intention. For example, in the case ofsuperimposing the second content on a first extension area by a userinput, the AR providing device may map the second content to the firstextension area of the display module 160 and provide the same. Theactive window may be divided corresponding to the number of pieces ofcontent to be displayed.

As will be described later, the affordance may be variable (e.g.,extended) by a user input (e.g., an affordance extension interaction).According to an embodiment of the disclosure, if one piece of content(e.g., the first content 1101) is disposed in (e.g., overlaps) theextended affordance, the AR providing device may extend the displaymodule 160 of the electronic device 101 to correspond to the extendedaffordance and configure an active window so as to display one piece ofcontent on the entire extended screen through the entire area of theextended extension area and the default area. According to anotherembodiment of the disclosure, if a plurality of pieces of content (e.g.,the first content 1101 and the second content 1102) is disposed in(e.g., overlaps) the extended affordance, the AR providing device mayconfigure an active window in each of the default area and the extensionarea so as to display the plurality of pieces of content in the defaultarea and the extension area, respectively, or configure at least oneactive window in the extension area so as to display the plurality ofpieces of content in the extension area corresponding to the extendedaffordance.

In operation 1019, the AR providing device provide content, based on theactive window of the target device. The AR providing device may providecontent to correspond to a first display mode or a second display modethrough the target device. For example, the AR providing device maydisplay content in a first display mode in which content is providedthrough an active window (e.g., a real display area of the electronicdevice 101) configure in the electronic device 101. As another example,the AR providing device may display the content in a second display modein which content is virtually mapped and provided through an activewindow configure based on the electronic device 101 in the AR screen ofthe wearable electronic device 301. Examples of this are shown in FIGS.11E and 11F.

FIG. 11E shows an example of providing content according to the firstdisplay mode. Referring to FIG. 11E, based on the determination of thefirst display mode, the AR providing device may perform processing todisplay the content 1110 selected and determined by the user on the ARscreen 1100, based on a substantial display area (e.g., an extensionarea extended by sliding-out) corresponding to the display module 160 ofthe electronic device 101. If the electronic device 101 performsprocessing of content or an application corresponding to the contentbeing executed in relation to AR services, the AR providing device mayoperate to stop displaying the current screen of the wearable electronicdevice 301 (e.g., turn off the display module 520 or the wearableelectronic device 301 in the state in which the wearable electronicdevice 301 is taken off) and continuously display the first content 1101selected and determined by the user among the content 1100, which isbeing displayed through the AR screen 1100, through the display module160 of the electronic device 101 in the real space.

The content (e.g., the first content 1101) displayed through the displaymodule 160 may be the same as the content (e.g., the first content 1101)being displayed on the AR screen 1100 or may be another piece of contentobtained by processing (e.g., resizing) the content being displayed onthe AR screen 1100 to correspond to the display area of the displaymodule 160. The content being displayed on the AR screen 1100 and thecontent to be moved to the electronic device 101 and displayed throughthe display module 160 may include substantially the same executionscreen (or content) and have continuity, and the display size thereofmay vary depending on the display area.

FIG. 11F shows an example of providing content according to the seconddisplay mode. Referring to FIG. 11F, based on the determination of thesecond display mode, the AR providing device may perform processing todisplay the content 1110 based on a virtual area associated with theelectronic device 101 on the AR screen 1100 provided through the displaymodule 520 of the wearable electronic device 301. If the wearableelectronic device 301 performs processing of content or an applicationcorresponding to the content being executed in relation to AR services,or if the content to be provided corresponds to a specified profile, theAR providing device may perform the following operations. For example,the AR providing device may operate to map (e.g., texture-map) the firstcontent 1101 selected and determined by the user, among the content 1100being displayed through the AR screen 1100, to the default area of theelectronic device 101 positioned in the real space displayed on the ARscreen 1100 through the display module 520 of the wearable electronicdevice 301, the extension area of the electronic device 101, and/or thevirtual area produced adjacent to the electronic device 101 and displaythe same thereon.

For example, the AR providing device may display the first content 1101selected and determined by the user through the entirety or at least aportion of the default area of the electronic device 101, the extensionarea thereof, and/or the virtual area in the AR screen 1100 (e.g., avirtual space) according to the second display mode, and maintain theexisting display state of the remaining unselected content (e.g., thesecond content 1102 and the third content 1103) on the AR screen 1100.

The content (e.g., the first content 1101) displayed in a specified areathrough mapping may be the same as the content (e.g., the first content1101) being displayed on the AR screen 1100 or may be another piece ofcontent obtained by processing (e.g., resizing) the content beingdisplayed on the AR screen 1100 to correspond to the specified area. Thecontent being displayed on the AR screen 1100 and the content displayedthrough mapping may include substantially the same execution screen (orcontent) and have continuity, and the display size thereof may varydepending on the display area.

An element 1190 may show an example of a display state of the displaymodule 160 of the electronic device 101 viewed in the real space,instead of the electronic device 101 on the AR screen 1100 providedthrough the display module 520 of the wearable electronic device 301. Asillustrated, the electronic device 101 may maintain a screen displaybeing executed (or operated) in the electronic device 101, as shown inFIG. 11A, through a default area and provide a blank screen or aspecified screen to an extension area 1195 corresponding to extension ofthe display module 160 (e.g., an area extended by sliding-out).

For example, the electronic device 101 may provide, as a specifiedscreen, a screen resulting from execution of a specified function inresponse to extension of the display module 160 (e.g., a split screenbased on execution of a specified application or the entire screen of arunning application (e.g., display the screen displayed in the defaultarea into a full screen)). The display module 160 of the electronicdevice 101 in the real space may be in the state of being turned off.

For example, in the second display mode, the screen (e.g., the displayscreen of the first content 1101) displayed based on the display module160 of the electronic device 101 on the AR screen 1100 and the screendisplayed through the display module 160 of the electronic device 101 inthe real space may be different, and the display module 160 of theelectronic device 101 may not display the content (e.g., the firstcontent 1101) on the AR screen 1200 in the real space.

FIGS. 12A to 12F are diagrams illustrating an example of operationscenarios of an electronic device or a wearable electronic deviceaccording to various embodiments of the disclosure.

FIGS. 12A to 12F show examples of changing a display area for providingcontent according to a user input, in addition to the operationdescribed with reference to FIGS. 10, and 11A to 11F.

Referring to FIG. 12A, an AR providing device (e.g., an electronicdevice 101 or a wearable electronic device 301) may provide content(e.g., first content 1201, second content 1202, and third content 1203)to an AR screen 1200 through the display module 520 of the wearableelectronic device 301. The wearable electronic device 301 may be worn onthe user, and the AR providing device may display a portion of content(or virtual information) superimposed on at least a portion of a realspace on the AR screen 1200 through the display module 520 of thewearable electronic device 301. For example, the AR providing device maysuperimpose at least one piece of content (e.g., first content 1201,second content 1202, and third content 1203) in the real spacecorresponding to the field of view of the user, thereby providing thecontent to the user.

In the above situation, the user may move an electronic device 101 intothe AR screen 1200 provided through the display module 520 of thewearable electronic device 301. If the AR providing device detects auser's specified interaction (e.g., a rollable activation interaction)for the electronic device 101 on the AR screen 1200, the AR providingdevice may provide an affordance (or guide) 1220 for a virtual area(e.g., the area expected by rollable extension) in response to thespecified interaction. The specified interaction may include, forexample, a slide-out or rollable extension gesture, an input to aspecified hardware button for an extension trigger, or an input to aspecified software button for an extension trigger. The AR providingdevice may provide a virtual window guide corresponding to a defaultarea and/or a substantial extension area of the electronic device 101 asthe affordance 1220. The provision of the affordance 1220 according toan embodiment will be described with reference to the drawings to bedescribed later.

The user may move the electronic device 101 such that the affordance1220 at least partially overlaps the first content 1201 on the AR screen1200. The AR providing device may track the electronic device 101 in theAR screen 1200 (e.g., track an object) and move the affordance 1220together in response to the movement of the electronic device 101. Inresponse to the movement of the electronic device 101 by the user, theaffordance 1220 and the first content 1201 may overlap each other on theAR screen 1200. If the affordance 1220 and the first content 1201overlap each other, the AR providing device may activate the overlappingfirst content 1201 (e.g., move and display the content forward of theaffordance 1220), thereby providing the overlapping first content 1201and the remaining non-overlapping content (e.g., the second content 1202and the third content 1203) to be distinguished. For example, the ARproviding device may highlight and provide the overlapping first content1201 to be associated with the affordance 1220 so that the user mayintuitively recognize the overlapping first content 1201.

The electronic device 101 may be a bar-type electronic device in whichthe display module 160 is not extended and has a fixed size, or may be arollable type electronic device in which the display module 160 isextendable. For example, the electronic device 101 may include asmartphone, a tablet PC, a notebook computer, and/or a display device(e.g., a monitor or TV) including a display module 160 having a fixedsize or a variable size.

According to an embodiment of the disclosure, if the electronic device101 is a bar-type electronic device, an affordance may be provided basedon a display area (e.g., a default area) of the display module 160 ofthe electronic device 101, and the affordance and content may be mappedin the display area.

According to another embodiment of the disclosure, if the electronicdevice 101 is a rollable type electronic device and if the displaymodule 160 of the electronic device 101 is in an extended state, anaffordance may be provided based on a display area (e.g., default area+extension area) of the display module 160, and the affordance andcontent may be mapped in the display area.

If the electronic device 101 is a rollable type electronic device and ifthe display module 160 of the electronic device 101 is in a non-extendedstate, the display module 160 may provide an affordance, based on anextendable virtual area (e.g., a virtual area having a sizecorresponding to the extension area), and the affordance and content maybe mapped in the virtual area.

The content in the AR screen provided through the display module 520 ofthe wearable electronic device 301 may include space-oriented contentthat is mapped to the real space and the position of which is adaptivelymovable according to the user's gaze or the user's movement (e.g., headmovement) (e.g., the content is moved to always be visible within theuser's gaze, that is, fixedly displayed in a specified area in thedisplay module 520). As another example, the content in the AR screenprovided through the display module 520 of the wearable electronicdevice 301 may include device-oriented (or object-oriented) content thatis not mapped to the real space but fixedly mapped to a specific space(or an area where a corresponding object is located in the real space)on the AR screen of the display module 520 so as not to be movableaccording to the user's gaze or the user's movement.

An operation of mapping (e.g., overlapping) the affordance and thecontent may vary depending on, for example, whether the content isspace-oriented content or device-oriented content. For example, in thecase of space-oriented content, the user may overlap content and theaffordance by directly moving the electronic device 101 within the ARscreen of the display module 520 of the wearable electronic device 301.As another example, in the case of device-oriented content, the user maymove the user's gaze (e.g., move the head) to the display module 160(e.g., the default area, the extension area, the entire area, or thevirtual area) of the electronic device 101 within the AR screen of thedisplay module 520 of the wearable electronic device 301, and thecontent may be moved on the AR screen in response to the movement of theuser's gaze so that the content may overlap the affordance in thecorresponding area of the electronic device 101.

The movement of content on the AR screen is not limited to the user'sgaze and may be performed, for example, by a user gesture on the ARscreen (e.g., a user's gesture of selecting and moving content on the ARscreen). In addition to the selection by overlapping of the firstcontent 1201, the user may input a first interaction 1230 (e.g., anaffordance extension interaction) for varying (e.g., extending) theaffordance 1220 in order to further select another piece of content(e.g., the second content 1202) by overlapping of the affordance 1220.In an embodiment, the AR providing device may extend the affordance 1220in response to the first interaction and provide the same. An example ofthis is shown in FIG. 12B.

Although it is described that the affordance 1220 is extended (e.g.,varied) based on the user's interaction, this example is intended onlyto help the understanding of the disclosure and is not intended to limitembodiments of the disclosure. For example, the affordance 1220, is notlimited to the variation based on the user's interaction, and mayautomatically vary. For example, if the size of the extension area bythe affordance 1220 is not suitable for displaying content (e.g.,excessively larger or smaller than the displayable size of the content,)in displaying the content overlapping the affordance 1220, the ARproviding device may calculate the size of the extension area capable ofdisplaying the overlapping content and vary (e.g., extend) theaffordance 1220 to conform to the size of the calculated extension area,thereby providing the same.

Referring to FIG. 12B, the AR providing device may vary (e.g., extend)the affordance 1220 to conform to the variable size (or movementdistance) of the first interaction and provide the same to the AR screen1200. As shown in FIG. 12B, as the affordance 1220 is extended, anotherpiece of content (e.g., the second content 1202), as well as the firstcontent 1201, may overlap the affordance 1220. If the affordance 1220and the second content 1202 overlap in addition to the first content1201, the AR providing device may further activate the overlappingsecond content 1202 (e.g., move and display the content forward of theaffordance 1220), thereby providing the overlapping first content 1201and second content 1202 to be distinguished from the remainingnon-overlapping content (e.g., the third content 1203). For example, theAR providing device may highlight and provide the overlapping firstcontent 1201 and second content 1202 to be associated with theaffordance 1220 so that the user may intuitively recognize theoverlapping first content 1201 and second content 1202.

Referring to FIG. 12C, the user may move the electronic device 101 inorder to further select another piece of content (e.g., the thirdcontent 1203) in addition to the selection by the overlapping of thefirst content 1201 and second content 1202. According to an embodiment,in order to select all of the content (e.g., the first content 1201, thesecond content 1202, and the third content 1203) disposed side by sideon the AR screen 1200, the user may move the electronic device 101 suchthat the affordance 1220 at least partially overlaps the third content1203 on the AR screen 1200.

Referring to the example in FIG. 12C, the electronic device 101 may movein the right direction, and the affordance 1220 may also move inresponse to the movement of the electronic device 101. The first content1201, the second content 1202, and the third content 1203 may be fixedto be displayed without changing their positions. An overlapping portionof the first content 1201, the second content 1203, and the thirdcontent 1205 may vary in response to the movement of the electronicdevice 101. For example, the first content 1201 may be positioned on thedisplay module 160 of the electronic device 101, the second content 1202may be positioned in the area where the first content 1201 waspositioned in the affordance 1220 by replacing the same, and the thirdcontent 1203 may be further positioned in the area where the secondcontent 1202 was positioned in the affordance 1220.

Referring to FIG. 12D, the AR providing device may detect a secondinteraction 1240 of triggering an operation (e.g., an operation trigger)of displaying at least one piece of content (e.g., the first content1201, the second content 1202, or the third content 1203) selected bythe user through a specified target device. For example, the user mayselect and determine the first content 1201, the second content 1202,and the third content 1203 on the AR screen 1200 and input a specifiedsecond interaction 1240 (e.g., a rollable extension gesture, anotherspecified touch gesture, or a voice-based command) for displaying theselected first content 1201, second content 1202, and third content 1203through the target device.

The AR providing device may determine a target device for displayingcontent, based on the detection of the second interaction 1240. The ARproviding device may determine a display mode (e.g., a first displaymode or a second display mode) corresponding to a target device fordisplaying the content selected by the user. For example, the ARproviding device may determine a display mode and a target devicecorresponding thereto, based on a content processing entity and/or aspecified profile of the content.

The AR providing device may provide content through the target device toconform to the first display mode or the second display mode. Forexample, the AR providing device may display content in a first displaymode in which content is provided through a real display area based onthe display module 160 of the electronic device 101. As another example,the AR providing device may display content in a second display mode inwhich content is mapped and provided through a virtual display areaconfigure based on the electronic device 101 in the AR screen 1200 ofthe wearable electronic device 301. Examples of this are shown in FIGS.12E and 12F.

FIG. 12E shows an example of providing content according to the firstdisplay mode.

Referring to FIG. 12E, based on the determination of the first displaymode, the AR providing device may perform processing to display thecontent (e.g., the first content 1201, the second content 1202, and thethird content 1203) selected and determined by the user on the AR screen1200, based on a substantial display area (e.g., a default area and anextension area that is extended by sliding-out) corresponding to thedisplay module 160 of the electronic device 101.

If the electronic device 101 performs processing of content or anapplication corresponding to the content being executed in relation toAR services, the AR providing device may operate to stop displaying thecurrent screen of the wearable electronic device 301 (e.g., turn off thedisplay module 520 or the wearable electronic device 301 in the state inwhich the wearable electronic device 301 is taken off) and continuouslydisplay the first content 1201, the second content 1202, and the thirdcontent 1203 selected and determined by the user on the AR screenthrough the display module 160 of the electronic device 101 in the realspace.

The AR providing device may display the content on the display module160 to conform to the position where each piece of content 1201, 1202,or 1203 overlaps the electronic device 101 on the AR screen 1200. Forexample, the first content 1201 may overlap the default area of theelectronic device 101 on the AR screen 1200, and the first content 1201may be provided through the default area of the display module 160 ofthe electronic device 101 in the real space. For example, the secondcontent 1202 may overlap the first extension area of the electronicdevice 101 on the AR screen 1200, and the second content 1202 may beprovided through the first extension area of the display module 160 ofthe electronic device 101 in the real space. For example, the thirdcontent 1203 may overlap the second extension area of the electronicdevice 101 on the AR screen 1200, and the third content 1203 may beprovided through the second extension area of the display module 160 ofthe electronic device 101 in the real space.

FIG. 12F shows an example of providing content according to the seconddisplay mode.

Referring to FIG. 12F, based on the determination of the second displaymode, the AR providing device may perform processing to display thecontent (e.g., the first content 1201, the second content 1202, and thethird content 1203), based on a virtual area associated with theelectronic device 101 on the AR screen 1200 provided through the displaymodule 520 of the wearable electronic device 301.

If the wearable electronic device 301 performs processing of content oran application corresponding to the content being executed in relationto the AR service, or if the content to be provided corresponds to aspecified profile, the AR providing device may perform the followingoperations. For example, the AR providing device may operate to map(e.g., texture-map) the first content 1201, the second content 1202, andthe third content 1203 selected and determined by the user from the ARscreen 1200 to the default area of the electronic device 101 positionedin the real space displayed on the AR screen 1200 through the displaymodule 520 of the wearable electronic device 301, the extension area ofthe electronic device 101, and/or the virtual area produced adjacent tothe electronic device 101 and display the same thereon. The AR providingdevice may display the first content 1201, the second content 1202, andthe third content 1203 selected and determined by the user through theentirety or at least a portion of the default area of the electronicdevice 101, the extension area thereof, and/or the virtual area in theAR screen 1200 (e.g., a virtual space) according to the second displaymode.

An element 1290 may show an example of a display state of the displaymodule 160 of the electronic device 101 viewed in the real space,instead of the electronic device 101 on the AR screen 1200 providedthrough the display module 520 of the wearable electronic device 301. Asillustrated, the electronic device 101 may maintain a screen displaybeing executed (or operated) in the electronic device 101, as shown inFIG. 12A, through a default area and provide a blank screen or aspecified screen to an extension area 1295 corresponding to extension ofthe display module 160 (e.g., an area extended by sliding-out). Forexample, the electronic device 101 may provide, as a specified screen, ascreen resulting from execution of a specified function in response toextension of the display module 160 (e.g., a split screen based onexecution of a specified application or the entire screen of a runningapplication (e.g., display the screen displayed in the default area intoa full screen)). According to an embodiment, the display module 160 ofthe electronic device 101 in the real space may be in the state of beingturned off.

In the second display mode, the screen (e.g., the display screen of thefirst content 1201, the second content 1202, and the third content 1203)displayed based on the display module 160 of the electronic device 101on the AR screen 1200 and the screen displayed through the displaymodule 160 of the electronic device 101 in the real space may bedifferent, and the display module 160 of the electronic device 101 maynot display the content (e.g., the first content 1201, the secondcontent 1202, and the third content 1203) on the AR screen 1200 in thereal space.

The AR providing device may dispose and display the content tocorrespond to the position where each piece of content 1201, 1202, or1203 overlaps the electronic device 101 on the AR screen 1200. Forexample, the first content 1201 may overlap the default area of theelectronic device 101 on the AR screen 1200, and the first content 1201may be mapped and provided to the default area of the display module 160of the electronic device 101. The second content 1202 may overlap thefirst extension area (or first virtual area) of the electronic device101 on the AR screen 1200, and the second content 1202 may be mapped andprovided to the first extension area (or first virtual area) of thedisplay module 160 of the electronic device 101. The third content 1203may overlap the second extension area (or second virtual area) of theelectronic device 101 on the AR screen 1200, and the third content 1203may be mapped and provided to the second extension area (or secondvirtual area) of the display module 160 of the electronic device 101.

FIGS. 13A to 13C are diagrams illustrating an example of configuring anactive window in association with an electronic device provided throughan AR screen according to various embodiments of the disclosure.

FIGS. 13A, 13B, and 13C show various examples of providing an affordancefor an expected screen in which the content selected by the user on anAR screen 1300 is to be displayed according to a first display mode or asecond display mode to the AR screen 1300. An active window may beconfigured to correspond to a given affordance in the first display modeor the second display mode. For example, according to the disclosure, itis possible to guide the user in advance about an expected screen inwhich the content selected by the user on the AR screen 1300 is to bedisplayed in a corresponding display mode using affordance.

Referring to FIG. 13A, an affordance 1310 for an active window may beprovided to a virtual area V while a default area A of the electronicdevice 101 is maintained. The virtual area V may be produced accordingto the size (or ratio) corresponding to the default area A or may beproduced according to the size (or ratio) corresponding to theextendable extension area of the electronic device 101. If the size ofthe extension area by the affordance 1310 is not suitable for displayingcontent (e.g., excessively larger or smaller than the displayable sizeof the content,) in displaying the content, the AR providing device maycalculate the size of the extension area capable of displaying theoverlapping content and vary (e.g., extend) the affordance 1310 toconform to the size of the calculated extension area so as to providethe content. Accordingly, the virtual area V may vary depending on thesize (or ratio) of the variable affordance 1310.

Referring to FIG. 13B, since a user interaction (e.g., a rollableactivation gesture) reflects a clear intention of the user for screenextension of the electronic device 101, an affordance 1320 for an activewindow may be provided to the entire area including the default area Aof the electronic device 101 and the virtual area V. For example, theentire area (e.g., default area +extension area) including the defaultarea and the extension area of the display module 160 of the electronicdevice 101 may be configured as an active window for displaying content.

Referring to FIG. 13C, in FIG. 13C, an affordance 1330 for an activewindow may be provided based on an extension area E in which theelectronic device 101 is substantially extended on the AR screen 1300.For example, if the display module 160 of the electronic device 101 issubstantially extended, the extension area E may be configured as anactive window for displaying content.

Referring to FIGS. 13A to 13C, affordances 1310, 1320, and 1330 may bevaried (e.g., extended) by a user input (e.g., an affordance extensioninteraction). According to an embodiment of the disclosure, if one pieceof content (e.g., first content) is disposed in (e.g., overlaps) theextended affordance, the AR providing device may extend the displaymodule 160 of the electronic device 101 to correspond to the extendedaffordance and configure an active window so as to display one piece ofcontent on the entire extended screen through the entire area of theextended extension area and the default area. According to anotherembodiment of the disclosure, if a plurality of pieces of content (e.g.,first content and second content) is disposed in (e.g., overlaps) theextended affordance, the AR providing device may configure an activewindow in each of the default area and the extension area so as todisplay the plurality of pieces of content in the default area and theextension area, respectively, or configure at least one active window inthe extension area so as to display the plurality of pieces of contentin the extension area corresponding to the extended affordance.

When configuring an active window, the AR providing device may configurethe active window in at least one of the default area, the extensionarea, or the entire area, based on the overlapping state (or overlappingarea) of the affordance and the content according to the user'sintention.

For example, if the content overlaps the affordance, the AR providingdevice may configure an active window to display the content in theextension area. As another example, if the content overlaps both theelectronic device 101 (e.g., the default area of the electronic device101) and the affordance on the AR screen, the AR providing device mayconfigure an active window to display the corresponding content in theentire area (e.g., default area+extension area (or virtual area)). Asanother example, if a plurality of pieces of content (e.g., firstcontent and second content) overlaps the affordance, the AR providingdevice may configure an active window in each of the default area andthe extension area to display the plurality of pieces of content in thedefault area and the extension area, respectively.

As another example, if one piece of content (e.g., first content) amonga plurality of pieces of content overlaps the affordance and anotherpiece of content (e.g., second content) thereof overlaps the electronicdevice 101 (e.g., the default area of the electronic device 101), forexample, if each piece of content is disposed according to the user'sintention (e.g., a user input of selecting and moving content to acertain area), the AR providing device may configure an active windowfor the first content in the extension area and an active window for thesecond content in the default area so as to display the content toconform to the disposition according to the user's intention. In thecase of superimposing the second content on a first extension area by auser input, the AR providing device may map the second content to thefirst extension area of the display module 160 and provide the secondcontent. The active window may be divided corresponding to the amount ofcontent to be displayed.

FIGS. 14A to 14F are diagrams illustrating an example of operationscenarios of an electronic device or a wearable electronic deviceaccording to various embodiments of the disclosure.

Referring to FIGS. 14A to 14F, the illustrated operations may beperformed by the wearable electronic device 301 according to thestandalone method of providing an AR service. For example, FIGS. 14A to14F show examples in which the electronic device 101 is unable toprocess content or an application corresponding to the content beingexecuted in relation to an AR service. FIGS. 14A to 14F show examples ofmapping and displaying content displayed on an AR screen providedthrough the display module 520 of the wearable electronic device 301 onthe AR screen in association with the electronic device 101 andprocessing a user input for the content displayed to be mapped to theelectronic device 101. FIGS. 14A to 14F include an example ofmanipulating content being executed through the wearable electronicdevice 301 in the standalone method using the electronic device 101after the display module 160 of the electronic device 101 is extended(e.g., slides out). In FIGS. 14A to 14F, content is implemented (orused) by the wearable electronic device 301 in the standalone method andmay be, for example, content that is unable to be processed by theelectronic device 101.

Referring to FIG. 14A, the wearable electronic device 301 may providecontent 1410 to an AR screen 1400 through the display module 520 of thewearable electronic device 301. The wearable electronic device 301 maybe worn on the user, and at least one piece of content (or virtualinformation) superimposed on at least a portion of a real space may bedisplayed on the AR screen 1400 through the display module 520 of thewearable electronic device 301. For example, the wearable electronicdevice 301 may superimpose at least one piece of content 1410 in thereal space corresponding to the field of view of the user, therebyproviding the same to the user.

Referring to FIG. 14B, in the above situation, the user may move theelectronic device 101 into the AR screen 1400 provided through thedisplay module 520 of the wearable electronic device 301.

Referring to FIG. 14C, if the wearable electronic device 301 detects auser's specified interaction (e.g., a rollable activation interaction)for the electronic device 101 on the AR screen 1400, the wearableelectronic device 301 may provide an affordance (or guide) 1420 for avirtual area (e.g., the area expected by rollable extension or the areacalculated to conform to the size of the content 1410) in response tothe specified interaction. The wearable electronic device 301 mayprovide a virtual window guide corresponding to a default area and/or asubstantial extension area of the electronic device 101 as theaffordance 1420.

The user may move the electronic device 101 such that the affordance1420 overlaps the content 1410 on the AR screen 1400. According to anembodiment, the wearable electronic device 301 may track the electronicdevice 101 in the AR screen 1400 (e.g., track an object), and theaffordance 1420 may also move in response to the movement of theelectronic device 101. In response to the movement of the electronicdevice 101 by the user, the affordance 1420 and the content 1410 mayoverlap each other on the AR screen 1400. If the affordance 1420 and thecontent 1410 overlap, the wearable electronic device 301 may activatethe overlapping content 1401 (e.g., move and display the content forwardof the affordance 1420), thereby providing the overlapping content 1401to be distinguished. For example, the wearable electronic device 301 mayhighlight and provide the overlapping content 1410 to be associated withthe affordance 1420 so that the user may intuitively recognize theoverlapping (or user-selected) content 1410.

The wearable electronic device 301 may vary (e.g., extend) theaffordance 1420, based on the variable size (or movement distance)corresponding to an additional interaction by the user, and provide thesame to the AR screen 1400. For example, the user may further extend theaffordance 1420 such that all of the content 1410 on the AR screen 1400is to be included in the affordance 1420, so that the content 1410 mayoverlap the affordance 1420 according to the additional extension of theaffordance 1420.

The AR providing device may determine the size of the affordance 1420capable of displaying the content 1410, based on the size of the content1410, and the additional extension of the affordance 1420 may beautomatically performed according to the determined size. For example,if the size of the extension area by the affordance 1420 is not suitablefor displaying content (e.g., excessively larger or smaller than thedisplayable size of the content,) in displaying the content, the size ofthe extension area capable of displaying the overlapping content 1410may be calculated, and the affordance 1420 may be varied (e.g.,extended) to conform to the size of the calculated extension area andthen provided. Accordingly, the extension area V of the electronicdevice 101 may vary according to the size (or ratio) of the variableaffordance 1420.

Referring to FIG. 14D, the wearable electronic device 301 may detect aninteraction 1430 of triggering an operation (e.g., an operation trigger)for displaying the content 1410 selected by the user through a specifiedtarget device. For example, the user may select and determine thecontent 1401 from the AR screen 1400 and input a specified interaction1430 (e.g., a rollable extension gesture, another specified touchgesture, or a voice-based command) for displaying the selected content1410 through the target device.

The wearable electronic device 301 may determine a target device fordisplaying the content 1410, based on the detection of the interaction1430. The AR providing device may determine a display mode (e.g., afirst display mode or a second display mode) corresponding to a targetdevice for displaying the content 1410 selected by the user. Forexample, the wearable electronic device 301 may determine a display modeand a target device corresponding thereto, based on a processing entityof the content 1410 and/or a specified profile of the content 1410.

The wearable electronic device 301 may provide content to correspond tothe first display mode or the second display mode through the targetdevice. For example, the wearable electronic device 301 may display thecontent 1410 in a first display mode in which content is providedthrough a real display area based on the display module 160 of theelectronic device 101. The content 1410 may include content implemented(or used) by the wearable electronic device 301 in the standalonemethod. For example, the content 1410 may indicate content that isunable to be processed by the electronic device 101. Therefore, whenproviding the content 1410 in the first display mode, the wearableelectronic device 301 may operate to mirror and provide the content 1410to the electronic device 101. As another example, the wearableelectronic device 301 may display the content 1410 in a second displaymode in which content is mapped and provided through a virtual displayarea configure based on the electronic device 101 in the AR screen 1400of the wearable electronic device 301. Examples of this are shown inFIGS. 14E and 14F.

FIG. 14E shows an example of providing content of the wearableelectronic device 301 through the electronic device 101 according to thefirst display mode. Referring to FIG. 14E, based on the determination ofthe first display mode, the wearable electronic device 301 may performprocessing to display the content 1410 selected and determined by theuser on the AR screen 1400, based on a substantial display area (e.g., adefault area and an extension area that is extended by sliding-out)corresponding to the display module 160 of the electronic device 101.

If the wearable electronic device 301 processes content or anapplication corresponding to the content is being executed in relationto AR services, the wearable electronic device 301 may operate to mirrorand transmit the content 1410 selected by the user from the AR screen1400 to the electronic device 101 and continuously display the contentthrough the display module 160 of the electronic device 101 in the realspace. For example, the wearable electronic device 301 may performprocessing of content or an application corresponding to the contentbeing executed in relation to AR services, and the electronic device 101may receive (e.g., mirroring) the content 1410 selected and determinedby the user on the wearable AR screen 1400 from the wearable electronicdevice 301 through specified communication and display the mirroredcontent 1410 through the display module 160 of the electronic device101.

The wearable electronic device 301 may transmit the content 1410 of thedisplay module 520 that is being processed to the electronic device 101so as to be displayed on the display module 160 of the electronic device101. For example, a transmission entity and a processing entity of thecontent 1410 may be the wearable electronic device 301. According to anembodiment, the electronic device 101 may serve to display the content1410 or related information transmitted from the wearable electronicdevice 301.

The electronic device 101 may extend the display module 160 tocorrespond to the size of the content 1410 being processed on the ARscreen 1400 of the wearable electronic device 301 or the size of theaffordance 1420 and display the content 1410 received from the wearableelectronic device 301, based on the default area and the extension area.For example, the content 1410 may overlap the affordance 1420 having afurther extended size on the AR screen 1400, and in the real space, thedisplay module 160 of the electronic device 101 may be extended (slidesout) to correspond to the size of the affordance 1420 so that thecontent 1410 may be provided through the default area and the extensionarea or through the extension area having a corresponding size,excluding the default area. The size of the extension area of theelectronic device 101 may be determined based on a predetermined ratiospecified by the user through the affordance 1420 on the AR screen 1400.

The electronic device 101 may operate to detect a user input (orinteraction) 1440 for controlling a function related to the content 1410displayed through the display module 160 and process a functioncorresponding to the user input 1440. The electronic device 101 mayrecognize a point (e.g., a touch point) at which the user input 1440 isdetected in the content 1410 and transmit related information about thedetected point (e.g., coordinates and/or spatial information) to thewearable electronic device 303 such that the wearable electronic device301 processes a function related to the related information.

The wearable electronic device 301 may recognize the point (e.g., atouch point) at which the user input 1440 is detected in the electronicdevice 101, based on the related information received from theelectronic device 101, perform processing to execute a related functionbased on the recognition result, and transmit content data resultingfrom the processing to the electronic device 101. The electronic device101 may receive the processing result (e.g., content data) from thewearable electronic device 301, reflect the received processing resulton the content 1410 on the display module 160, and display a resultscreen thereof through the display module 160.

An example of manipulating the content being executed through thewearable electronic device 301 in the standalone method using theelectronic device 101 after the display module 160 of the electronicdevice 101 is extended (e.g., slides out) will be described withreference to the drawings to be described later.

FIG. 14F shows an example of providing content according to the seconddisplay mode.

Referring to FIG. 14F, based on the determination of the second displaymode, the wearable electronic device 301 may perform processing todisplay the content 1410, based on a virtual area associated with theelectronic device 101, on the AR screen 1400 provided through thedisplay module 520 of the wearable electronic device 301. If thewearable electronic device 301 processes content or an applicationcorresponding to the content is being executed in relation to ARservices, or if the content to be provided corresponds to a specifiedprofile, the wearable electronic device 301 may perform the followingoperations. The wearable electronic device 301 may operate to map (e.g.,texture-map) the content 1410 selected and determined by the user on theAR screen 1400 to the default area of the electronic device 101positioned in the real space displayed on the AR screen 1400 through thedisplay module 520 of the wearable electronic device 301, the extensionarea of the electronic device 101, and/or the virtual area producedadjacent to the electronic device 101 and display the same thereon. Thewearable electronic device 301 may display the content 1410 selected anddetermined by the user through the entirety or at least a portion of thedefault area of the electronic device 101, the extension area thereof,and/or the virtual area in the AR screen 1400 (e.g., a virtual space)according to the second display mode.

The wearable electronic device 301 may disposed and display the content1410 in a virtual area having a size corresponding to the affordance1420 associated with the electronic device 101 on the AR screen 1400.For example, the content 1410 may overlap a virtual area correspondingto the affordance 1420 in the AR screen 1400, and the AR providingdevice may produce an active window (e.g., a virtual area) having acorresponding size based on the affordance 1420 and map the content 1410to the active window, thereby providing the same.

Sn element 1490 may show an example of a display state of the displaymodule 160 of the electronic device 101 viewed in the real space,instead of the electronic device 101 on the AR screen 1400 providedthrough the display module 520 of the wearable electronic device 301. Asillustrated, the electronic device 101 may display a screen beingexecuted (or operated) in the electronic device 101 shown in FIG. 14Athrough a default area or the entire area or maintain the display in thedefault area and provide a blank screen or a specified screen to anextension area 1495 corresponding to extension of the display module 160(e.g., an area extended by sliding-out). For example, the electronicdevice 101 may provide, as a specified screen, a screen resulting fromexecution of a specified function in response to extension of thedisplay module 160 (e.g., a split screen based on execution of aspecified application or the entire screen of a running application(e.g., display the screen displayed in the default area into a fullscreen)). According to an embodiment, the display module 160 of theelectronic device 101 in the real space may be in the state of beingturned off.

For example, in the second display mode, the screen (e.g., the displayscreen of the first content 1410) displayed based on the display module160 of the electronic device 101 on the AR screen 1400 and the screendisplayed through the display module 160 of the electronic device 101 inthe real space may be different, and the display module 160 of theelectronic device 101 may not display the content (e.g., the firstcontent 1410) on the AR screen 1400 in the real space.

The wearable electronic device 301 may operate to track and detect auser input (or interaction) 1450 for controlling a function related tothe content 1410 on the AR screen 1400 and process a functioncorresponding to the user input 1450. When displaying the content 1410based on the second display mode, the wearable electronic device 301 inthe standalone method may perform processing to track and detect theuser input 1450, based on technologies such as gaze tracking and objectdetection, calculate a point (e.g., a touch point) where the user input1450 is detected in the content 1410, and execute a related function.

The wearable electronic device 301 may detect a user input (orinteraction) for the content 1410 on the AR screen 1400, and theprocessing of the user input may be performed in the wearable electronicdevice 301. The wearable electronic device 301 may receive informationrelated to a user input from the electronic device 101 and cause theelectronic device 101 to process a user input made by the electronicdevice 101.

FIGS. 15A to 15F are diagrams illustrating an example of operationscenarios of an electronic device or a wearable electronic deviceaccording to various embodiments of the disclosure.

Referring to FIGS. 15A to 15F, the illustrated operations may beperformed by the electronic device 101 or the wearable electronic device301 depending on the tethered method or the standalone method forproviding an AR service. The electronic device 101 or the wearableelectronic device 301 will be referred to as an AR providing device forconvenience of description.

FIGS. 15A to 15F show examples in which the content (e.g., protectedcontent or privacy content) according to a specified profile or usersetting (e.g., protection configuration by the user), among the contentdisplayed in an AR screen provided through the display module 520 of thewearable electronic device 301, is not displayed on the display module160 of the electronic device 101 in the real space. Referring to FIG.15A, the AR providing device (e.g., the electronic device 101 or thewearable electronic device 301) may provide content (e.g., first content1510 and second content 1520) to an AR screen 1500 through the displaymodule 520 of the wearable electronic device 301. The wearableelectronic device 301 may be worn on the user, and the AR providingdevice may display at least one piece of content (or virtualinformation) superimposed on at least a portion of a real space on an ARscreen 1500 through the display module 520 of the wearable electronicdevice 301. For example, the AR providing device may superimpose atleast one piece of content (e.g., first content 1510 and second content1520) in the real space corresponding to the field of view of the user,thereby providing the content to the user.

In the above situation, the user may move the electronic device 101 intothe AR screen 1500 provided through the display module 520 of thewearable electronic device 301. If the AR providing device detects aspecified interaction (e.g., a rollable activation interaction) for theelectronic device 101 on the AR screen 1500, the AR providing device mayprovide an affordance (or guide) 1530 for a virtual area (e.g., the areaexpected by rollable extension) in response to the specifiedinteraction. The AR providing device may provide a virtual window guidecorresponding to a default area and/or a substantial extension area ofthe electronic device 101 as the affordance 1530.

The user may move the electronic device 101 such that the affordance1530 at least partially overlaps the first content 1510 on the AR screen1500. The AR providing device may track the electronic device 101 in theAR screen 1500 (e.g., track an object) and move the affordance 1530together in response to the movement of the electronic device 101. Inresponse to the movement of the electronic device 101 by the user, theaffordance 1530 and the first content 1510 may overlap each other on theAR screen 1500. If the affordance 1530 and the first content 1510overlap each other, the AR providing device may activate the overlappingfirst content 1510 (e.g., move and display the content forward of theaffordance 1530), thereby providing the overlapping first content 1510and the remaining non-overlapping content (e.g., the second content1520) to be distinguished. For example, the AR providing device mayhighlight and provide the overlapping first content 1510 to beassociated with the affordance 1530 so that the user may intuitivelyrecognize the overlapping first content 1510.

In addition to the selection by overlapping of the first content 1510,the user may vary (e.g., extend) the affordance 1530 in order to selectanother piece of content (e.g., the second content 1520) by overlappingof the affordance 1530. The AR providing device may extend theaffordance 1530 in response to a user input and provide the affordance1530. An example of this is shown in FIG. 15B.

Referring to FIG. 15B, the AR providing device may vary (e.g., extend)the affordance 1530 and provide the affordance 1530 to the AR screen1500. As shown in FIG. 15B, as the affordance 1530 is extended, thefirst content 1510 and the second content 1520 may overlap theaffordance 1530. If the affordance 1530 and the second content 1520overlap in addition to the first content 1510, the AR providing devicemay further activate the overlapping second content 1520 (e.g., move anddisplay the content forward of the affordance 1530), therebyhighlighting the first content 1510 and second content 1520 to beassociated with the affordance 1530 and providing the first content 1510and the second content 1520.

In the above situation, the user may configure at least one piece ofcontent to be protected (or hidden). For example, as shown in FIG. 15B,the user may input a first interaction 1540 (e.g., a protectionconfiguration interaction) in association with the second content 1520to configure the second content 1520 to be protected. The firstinteraction 1540 on the content of the AR screen 1500 may include aspecified touch gesture (e.g., a long-press gesture) for protection ofspecified content, an interaction of calling and selecting a menu forprotection of specified content, or a voice-based command capable ofinstructing protection of specified content. The protected (or hidden)content may be released from the protection, based at least on an inputcorresponding to the interaction for protecting the content or anotherinteraction specified to release the protection.

The AR providing device may detect a first interaction 1540 and, basedon the detection of the first interaction 1520, provide a protectionobject 1550 representing (or informing) that the content (e.g., thesecond content 1520) was configured to be protected (or hidden) inassociation (or mapping) with the corresponding content. An example ofthis is shown in FIG. 15C.

FIG. 15B shows an example of configuring content (e.g., the secondcontent 1520) to be protected (or hidden), based on the firstinteraction 1540 by the user. With regard to the protection (or hiding)configuration of the content, if a profile of the content belongs to aspecified profile, the content may be automatically configured to beprotected (or hidden) without user intervention (e.g., input of thefirst interaction 1540). The AR providing device may identify whether ornot the content (e.g., the second content 1520) has a specified profileand, based on identifying that the content has a specified profile,automatically provide a protection object 1550 indicating (or informing)that the content (e.g., the second content 1520) was configured to beprotected (or hidden) in association (or mapping) with the content. Anexample of this is shown in FIG. 15C.

Referring to FIG. 15C, the AR providing device may display theprotection object 1550 in association with the second content 1520,which is the corresponding content, based on the detection of the firstinteraction 1540 or the detection of a specified profile. For example,the AR providing device may provide the protection object 1550 to beassociated with the second content 1520 so that the user may intuitivelyrecognize the protection-specified content among a variety of contentprovided on the AR screen 1500. The protection object 1550 may beprovided in various forms to intuitively indicate that the correspondingcontent is configured to be protected (or hidden) and is invisible inthe real space. For example, the protection object 1550 may includestatic or dynamic text and/or images, and may be provided in variousshapes. The protection object 1550 may be provided to at least partiallyoverlap the corresponding content.

Referring to FIG. 15D, the AR providing device may detect a secondinteraction 1560 of triggering an operation (e.g., an operation trigger)of displaying at least one piece of content (e.g., the first content1510 or the second content 1520) selected by the user through aspecified target device. For example, the user may select and determinethe first content 1510 and the second content 1520 on the AR screen 1500and input a specified second interaction 1560 (e.g., a rollableextension gesture, another specified touch gesture, or a voice-basedcommand) for displaying the selected first content 1510 and secondcontent 1520 through the target device.

The AR providing device may determine a target device for displayingcontent, based on the detection of the second interaction 1560. The ARproviding device may determine a display mode (e.g., a first displaymode or a second display mode) corresponding to a target device fordisplaying the content selected by the user. For example, the ARproviding device may determine a display mode and a target devicecorresponding thereto, based on a processing entity of the content.

The AR providing device may provide content to correspond to the firstdisplay mode or the second display mode through the target device. Forexample, the AR providing device may display content in a first displaymode in which the content is provided through a real display area basedon the display module 160 of the electronic device 101 in the realspace. As another example, the AR providing device may display contentin a second display mode in which the content is mapped and providedthrough a virtual display area configure based on the electronic device101 in the AR screen 1500 of the wearable electronic device 301.Examples of this are shown in FIGS. 15E and 15F.

FIG. 15E shows an example of providing content according to the firstdisplay mode. Referring to FIG. 15E, based on the determination of thefirst display mode, the AR providing device may perform processing todisplay the first content 1510 not configured to be protected and to notdisplay the second content 1520 configured to be protected (e.g.,protected content or hidden content), among the content (e.g., the firstcontent 1510 and the second content 1520) selected and determined by theuser on the AR screen 1500, based on a substantial display area (e.g., adefault area and an extension area that is extended by sliding-out)corresponding to the display module 160 of the electronic device 101.

According to the first display mode, the AR providing device may operateto continuously display the first content 1510 and the second content1520 selected and determined by the user on the AR screen 1500 throughthe display module 160 of the electronic device 101 in the real spacesuch that only the first content 1510 is displayed on the display module160, instead of displaying the second content 1520 configured to beprotected in the AR screen 1500.

Referring to FIG. 15E, the electronic device 101 may turn on and extendthe display module 160, display the first content 1510 in the area(e.g., the first extension area) for first content 1510, instead ofdisplaying the second content 1520 in the area (e.g., the secondextension area) for the second content 1520, among specified areas(e.g., the default area and/or the extension area) of the display module160, and provide the second extension area as an empty space 1525 or byinactivating (or turning off).

FIG. 15F shows an example of providing content according to the seconddisplay mode.

Referring to FIG. 15F, based on the determination of the second displaymode, the AR providing device may display the content (e.g., the firstcontent 1510 and the second content 1520), based on a virtual areaassociated with the electronic device 101, on the AR screen 1500provided through the display module 520 of the wearable electronicdevice 301. For example, the AR providing device may display both thefirst content 1510 not configured to be protected and the second content1520 configured to be protected (e.g., protected content or hiddencontent) according to the second display mode.

According to the second display mode, the AR providing device mayoperate to map (e.g., texture-map) the first content 1510 and the secondcontent 1520 selected and determined by the user from the AR screen 1500to the default area of the electronic device 101 positioned in the realspace displayed on the AR screen 1500, the extension area of theelectronic device 101, and/or the virtual area produced adjacent to theelectronic device 101 and display the same thereon.

Referring to FIG. 15F, the electronic device 101 may turn on and extendthe display module 160, display the first content 1510 in the area(e.g., the first extension area) for first content 1510, and display thesecond content 1520 in the area (e.g., the second extension area) forthe second content 1520, among specified areas (e.g., the default areaand/or the extension area) of the display module 160. For example, theAR providing device may map and display the first content 1510 and thesecond content 1520 selected and determined by the user through theentirety or at least a portion of the default area of the electronicdevice 101, the extension area thereof, and/or the virtual area withinthe AR screen 1500 (e.g., a virtual space) according to the seconddisplay mode.

The element 1590 may show an example of a display state of the displaymodule 160 of the electronic device 101 viewed in the real space,instead of the electronic device 101 on the AR screen 1500 providedthrough the display module 520 of the wearable electronic device 301. Asillustrated, the electronic device 101 may display a screen beingexecuted (or operated) in the electronic device 101 shown in FIG. 15Athrough a default area or the entire area or maintain the display in thedefault area and provide a blank screen or a specified screen to anextension area 1595 corresponding to extension of the display module 160(e.g., an area extended by sliding-out). For example, the electronicdevice 101 may provide, as a specified screen, a screen resulting fromexecution of a specified function in response to extension of thedisplay module 160 (e.g., a split screen based on execution of aspecified application or the entire screen of a running application(e.g., display the screen displayed in the default area into a fullscreen)). The display module 160 of the electronic device 101 in thereal space may be in the state of being turned off.

For example, in the second display mode, the screen (e.g., the displayscreen of the first content 1410) displayed based on the display module160 of the electronic device 101 on the AR screen 1500 and the screendisplayed through the display module 160 of the electronic device 101 inthe real space may be different, and the display module 160 of theelectronic device 101 may not display the content (e.g., the firstcontent 1410) on the AR screen 1500 in the real space.

The AR providing device may provide the display based on the electronicdevice 101 and the display based on the wearable electronic device 301to be different depending on whether or not the content (e.g., the firstcontent 1510 and the second content 1520) selected and determined on theAR screen 1500 is configured to be protected. For example, as shown inFIG. 15E, the content (e.g., the second content 1520) configured to beprotected may not be displayed through the display module 160 of theelectronic device 101 in the real space. For example, as shown in FIG.15F, the content (e.g., the second content 1520) configured to beprotected may be mapped and displayed onto the display module 160 of theelectronic device 101 only in the user's virtual space of the AR screen1500 so that only the user wearing the wearable electronic device 301 isable to see the same.

FIG. 16 is a diagram illustrating an operation method between anelectronic device and a wearable electronic device according to anembodiment of the disclosure.

Referring to FIG. 16 , the illustrated operations show an example ofproviding content through the electronic device 101 or the wearableelectronic device 301, based on a display mode (e.g., a first displaymode or a second display mode), in an environment in which theelectronic device 101 and the wearable electronic device 301 provide ARservices based on the tethered method.

Operations performed by the electronic device 101 to be described belowmay be executed by the processor 120 of the electronic device 101including various processing circuits and/or executable program elementsof the electronic device 101. Operations performed by the wearableelectronic device 301 to be described below may be executed by theprocessor 590 of the wearable electronic device 301 including variousprocessing circuits and/or executable program elements of the wearableelectronic device 301.

Referring to FIG. 16 , in operation 1601, the electronic device 101 andthe wearable electronic device 301 may be connected through a specifiedcommunication method. For example, the electronic device 101 and thewearable electronic device 301 may be connected (e.g., paired) by wiredor wireless communication. For example, the electronic device 101 andthe wearable electronic device 301 may be connected to each otherthrough short-range wireless communication such as Bluetooth, BluetoothLow Energy, Wi-Fi, Wi-Fi Direct, or ultra-wide band (UWB), or wiredcommunication such as USB. The electronic device 101 may include asmartphone, a tablet personal computer (PC), and/or a notebook computer.The wearable electronic device 301 may include AR glasses, smartglasses, or a head mounted display (HMD).

In operation 1603, the electronic device 101 (or the processor 120 ofthe electronic device 101) may execute a related application for an ARservice. For example, the electronic device 101 may execute anapplication, based on a user input for manipulating execution of theapplication in the electronic device 101 or reception of a request forinstructing execution of the application from the wearable electronicdevice 301.

In operation 1605, the electronic device 101 may transmit content to thewearable electronic device 301, based on the execution of theapplication. For example, the electronic device 101 may process contentfor an AR service. The electronic device 101 may transmit relatedcontent data to the wearable electronic device 301 through specifiedcommunication to display the content on the wearable electronic device301.

In operation 1607, the wearable electronic device 301 (or the processor590 of the wearable electronic device 301) may display the content on anAR screen. For example, the wearable electronic device 301 may be in thestate of being worn on the user's body (e.g., a face), and the contentreceived from the electronic device 101 may be superimposed on the realworld and displayed through the display module 520 as a single image(e.g., an AR screen). The wearable electronic device 301 may receivecontent (e.g., image data or AR images) from the electronic device 101and display the received content together with various objects in thereal world through the display module 520.

In operation 1609, the wearable electronic device 301 may produce ARinformation. For example, the AR information may include informationabout a target object recognized by the wearable electronic device 301on the AR screen, interaction information related to the user's actionaccording to the user's gesture input and/or eye tracking on the ARscreen, object information about an object recognized on the AR screen,position information (e.g., coordinates and/or spatial information)about an object recognized on the AR screen, and/or position informationabout content displayed on the AR screen.

The interaction may include a first interaction of recognizing theelectronic device 101 entering the real space of the AR screen providedthrough the display module 520 of the wearable electronic device 301, asecond interaction of recognizing a specified gesture (e.g., a slide-outgesture or a rollable extension gesture) input by the user using theelectronic device 101 after the entry of the electronic device 101, or athird interaction of recognizing a specified menu manipulation gesture(e.g., a touch input) input by the user through the electronic device101 onto the AR screen after the entry of the electronic device 101.

The second interaction may include a gesture of substantially extendingan extension area of the electronic device 101. In another embodiment,the second interaction may not be a substantial extension of theextension area of the electronic device 101. For example, the user mayinput only an extending gesture without substantially extending theelectronic device 101, and the wearable electronic device 301 mayrecognize the user's gesture to identify the second interaction. In anembodiment, the wearable electronic device 301 may detect a user'sgesture according to the second interaction and/or the thirdinteraction, based on the user hand detecting and/or hand tracking onthe AR screen.

In operation 1611, the wearable electronic device 301 may transmit ARinformation to the electronic device 101. For example, the wearableelectronic device 301 may transmit AR information to the electronicdevice 101 through specified communication. The AR information may betransmitted in real time, periodically, or based at least on the time ofobtaining new information according to various state changes on the ARscreen.

In operation 1613, the electronic device 101 may detect a specifiedinteraction. For example, the electronic device 101 may receive ARinformation from the wearable electronic device 301 and analyze thereceived AR information to detect a specified interaction. The specifiedinteraction may indicate an operation trigger for determining a displaymode for displaying the content. The electronic device 101 may identifya specified interaction (or trigger) in which the electronic device 101is positioned on the user's gaze, based on AR information, while thecontent is displayed on the AR screen through the wearable electronicdevice 301.

The electronic device 101, based on the AR information, may identify afirst interaction in which the electronic device 101 enters the realspace of the AR screen provided through the display module 520 of thewearable electronic device 301, a second interaction related to aspecified gesture (e.g., a slide-out gesture or a rollable extensiongesture) input by the user using the electronic device 101 after theentry of the electronic device 101, or a third interaction related to aspecified menu manipulation gesture (e.g., a touch input) input by theuser through the electronic device 101 onto the AR screen after theentry of the electronic device 101.

In operation 1615, the electronic device 101 may determine a displaymode of the content, based on the detection of a specified interaction.For example, the electronic device 101 may detect (or recognize) aspecified interaction and, in response to the detection of the specifiedinteraction, determine a display mode to display content to correspondto at least one specified condition. The electronic device 101 maydetermine a display mode for displaying content, based at least on aprocessing entity for processing content being executed in relation toan AR service and/or a specified profile.

If the electronic device 101 detects a specified interaction, theelectronic device 101 may determine a display mode (e.g., a firstdisplay mode or a second display mode) for displaying information oncontent depending on a processing entity for processing content beingexecuted in relation to an AR service. According to various embodimentsof the disclosure, the display mode may include a first display mode(e.g., including a mirroring-based display mode) of displaying contenton the AR screen through the display module 160 of the electronic device101 or a second display mode of mapping the content on the AR screen tothe electronic device 101 on the AR screen in a virtual space (e.g., anAR screen) through the display module 520 of the wearable electronicdevice 301 and displaying the same therein, depending on the processingentity of content.

For example, in the case of the tethered method, the processing entityof content or an application of the content may be the electronic device101, and the first display mode may be determined as the display mode ofthe content. Even in the tethered method, if the wearable electronicdevice 301 is able to perform processing (e.g., switching of aprocessing entity) according to the properties of content, or if atarget device to display content is explicitly designated as thewearable electronic device 301 according to the user's intention (e.g.,the AR environment continues to be used), the second display mode may bedetermined as the display mode of the content.

In the case of the standalone method, the processing entity of contentor an application of the content may be the wearable electronic device301, and the second display mode may be determined as the display modeof the content. For example, even in the standalone method, if theelectronic device 101 is able to perform processing (e.g., switching ofa processing entity) according to the properties of content, or if atarget device to display content is explicitly designated as theelectronic device 101 according to the user's intention (e.g., mirroringor substantial extension of the display module 160 of the electronicdevice 101 in the real space), the first display mode may be determinedas the display mode for displaying the content.

The display mode determined according to the processing entity may varydepending on whether a profile of the content corresponds to a profilespecified in advance. For example, if the content corresponds to aspecified profile, a display mode (e.g. the second display mode) using avirtual space for the corresponding content (e.g., content of thespecified profile) may be determined, regardless of whether theprocessing entity is the electronic device 101 or the wearableelectronic device 301.

In operation 1617, based on the determination result, the electronicdevice 101 may determine whether the display mode is the first displaymode or the second display mode. Based on the determined display mode(e.g., the first display mode or the second display mode), theelectronic device 101 may operate to display the content through thedisplay module 160 of the electronic device 101 (e.g., the first displaymode) or display the content on an AR screen through the display module520 of the wearable electronic device 301 (e.g., the second displaymode).

If the display mode is determined to be the first display mode inoperation 1617, the electronic device 101 may configure an activewindow, based on a specified area of the display module 160 of theelectronic device 101, in operation 1621. For example, the electronicdevice 101 may determine an area corresponding to the position and/orsize determined for the content on the AR screen, among a substantialdisplay area (e.g., a default area and/or an extension area extendedaccording to sliding-out) in the display module 160, as a specified areafor displaying the content. The electronic device 101 may identify aspecified area in the display module 160 (e.g., identify the defaultarea and/or the extension area) and configure the specified area as anactive window for content.

If the display mode is determined to be the first display mode, theelectronic device 101 may operate to provide information indicatingoperation in the first display mode to the wearable electronic device301 and display the same through the wearable electronic device 301. Forexample, based on the determination of the first display mode, theelectronic device 101 may operate to display, through the wearableelectronic device 301, a guide stating that the content is to bedisplayed through the electronic device 101 due to switching to thefirst display mode so that the wearable electronic device 301 is to betaken off, thereby providing the same to the user.

Based on the determination of the first display mode, the electronicdevice 101 may transmit a related command instructing to stop displayingthe screen of the wearable electronic device 301 (e.g., turn off thedisplay module 520 or the wearable electronic device 301) to thewearable electronic device 301 together with the guide or subsequentlythereto. For example, the wearable electronic device 301 may operate toreceive a guide and a control command from the electronic device 101,display the guide for a predetermined period of time, and then stopdisplaying the screen according to the control command As anotherexample, the wearable electronic device 301 may operate to automaticallystop displaying the screen if it is taken off by the user.

In operation 1623, the electronic device 101 may provide (e.g., display)the content through the active window in the display module 160.

In operation 1625, the electronic device 101 may process a userinteraction (e.g., a user's touch input based on the displayed content)input through the electronic device 101. The electronic device 101 mayoperate to detect a user interaction for controlling a function relatedto the content displayed through the display module 160 and process afunction corresponding to the user interaction. The electronic device101 may recognize a point (e.g., a touch point) where a user interactionis detected in the content, process the user interaction to execute arelated function, based on the recognized result (e.g., touch inputcoordinates), and display content according to the processing resultthrough the display module 160.

If the display mode is determined to be the second display mode inoperation 1617, the electronic device 101 may map the active window,based on the electronic device 101 of the AR screen, in operation 1631.For example, the electronic device 101 may identify the state of theelectronic device 101 on the AR screen (e.g., position, whether it isextended, and/or whether it is virtually extended), based on the ARinformation, and determine information (e.g., a position and/or size) ofa virtual area for displaying the content on the AR screen, based on theidentification result. According to an embodiment, the electronic device101 may map the active window, based on the determined information onthe virtual area.

In operation 1633, the electronic device 101 may transmit mappinginformation to the wearable electronic device 301. For example, theelectronic device 101 may produce mapping information related to mappingof an active window and transmit the mapping information to the wearableelectronic device 301 through specified communication.

In operation 1635, the wearable electronic device 301 may providecontent through the active window on the AR screen. For example, thewearable electronic device 301 may receive mapping information from theelectronic device 101 and, based on the mapping information, map theactive window, based on the electronic device on the AR screen. Thewearable electronic device 301 may operate to map and display thecontent, based on the active window of the AR screen.

In operation 1637, the wearable electronic device 301 may process a userinteraction input onto the AR screen (e.g., a user's gesture input basedon the content displayed on the AR screen). The wearable electronicdevice 301 may track and detect a user interaction for controlling afunction related to the content on the AR screen. The wearableelectronic device 301 may track and detect the user input, based ontechnologies such as gaze tracking and object recognition, calculate apoint (e.g., a touch point) where the user interaction is detected inthe content on the AR screen, and transmit related information (orinteraction input information) (e.g., coordinates and/or spatialinformation) to the electronic device 101.

The electronic device 101 may recognize a point (e.g., a touch point)where a user interaction is detected in the content of the AR screen,based on the related information received from the wearable electronicdevice 301, perform processing to execute a related function, based onthe recognized result, and transmit content data according to theprocessing result to the wearable electronic device 301. The wearableelectronic device 301 may receive the content data from the electronicdevice 101 and reflect the received content data to content on the ARscreen, thereby displaying a result screen thereof (e.g., a screen inwhich the content data is mapped to the electronic device 101 on the ARscreen).

The wearable electronic device 301 may perform a user interaction forthe content on the AR screen by the electronic device 101 even in thesecond display mode. For example, the user may perform user interactionthrough the electronic device 101 in the real space even in the seconddisplay mode, and the electronic device 101 may process a functioncorresponding to the user interaction and may transmit related contentdata to the wearable electronic device 301 through specifiedcommunication connecting the wearable electronic device 301 and theelectronic device 101. The wearable electronic device 301 may operate toreceive and display the content data processed according to the userinteraction from the electronic device 101, thereby providing aprocessing result of the user interaction in the electronic device 101.

FIG. 17 is a diagram illustrating an operation method between anelectronic device and a wearable electronic device according to anembodiment of the disclosure.

Referring to FIG. 17 , the illustrated operations show an example ofproviding content through the electronic device 101 or the wearableelectronic device 301, based on a display mode (e.g., a first displaymode or a second display mode), in an environment in which the wearableelectronic device 301 provide AR services based on the standalonemethod.

Operations performed by the wearable electronic device 301 to bedescribed below may be executed by the processor 590 of the wearableelectronic device 301 including various processing circuits and/orexecutable program elements of the wearable electronic device 301.Operations performed by the electronic device 101 to be described belowmay be executed by the processor 120 of the electronic device 101including various processing circuits and/or executable program elementsof the electronic device 101.

Referring to FIG. 17 , in operation 1701, the electronic device 101 andthe wearable electronic device 301 may be connected through a specifiedcommunication method. For example, the electronic device 101 and thewearable electronic device 301 may be connected (e.g., paired) by wiredor wireless communication. The electronic device 101 and the wearableelectronic device 301 may be connected to each other through short-rangewireless communication such as Bluetooth, Bluetooth Low Energy, Wi-Fi,Wi-Fi Direct, or ultra-wide band (UWB), or wired communication such asUSB. The electronic device 101 may include a smartphone, a tablet PC,and/or a notebook computer. The wearable electronic device 301 mayinclude AR glasses, smart glasses, or a head mounted display (HMD).

In operation 1703, the wearable electronic device 301 (or the processor590 of the wearable electronic device 301) may execute an AR service.For example, the wearable electronic device 301 may execute anapplication related to an AR service, based on a user input formanipulating execution of the application in the wearable electronicdevice 301 or reception of a request for instructing execution of an ARservice from the electronic device 101.

In operation 1705, the wearable electronic device 301 may displaycontent on the AR screen, based on the execution of the AR service. Forexample, the wearable electronic device 301 may process content for anAR service. According to an embodiment, the wearable electronic device301 may be in the state of being worn on the user's body (e.g., a face)and display content (e.g., image data or AR images) through the displaymodule 520 together with various objects in the real world.

In operation 1707, the wearable electronic device 301 may detect aspecified interaction. The specified interaction may indicate anoperation trigger for determining a display mode for displaying content.The wearable electronic device 301 may identify a specified interaction(or trigger) in which the electronic device 101 is positioned on theuser's gaze while the content is displayed on the AR screen. Forexample, the wearable electronic device 301 may identify a firstinteraction in which the electronic device 101 enters the real space ofthe AR screen provided through the display module 520 of the wearableelectronic device 301, a second interaction related to a specifiedgesture (e.g., a slide-out gesture or a rollable extension gesture)input by the user using the electronic device 101 after the entry of theelectronic device 101, or a third interaction related to a specifiedmenu manipulation gesture (e.g., a touch input) input by the userthrough the electronic device 101 onto the AR screen after the entry ofthe electronic device 101.

In operation 1709, the wearable electronic device 301 may determine adisplay mode of content, based on the detection of the specifiedinteraction. For example, the wearable electronic device 301 may detect(or recognize) a specified interaction and, in response to the detectionof the specified interaction, determine a display mode for displayingcontent corresponding to at least one specified condition. The wearableelectronic device 301 may determine a display mode for displayingcontent, based at least on a processing entity for processing contentbeing executed in relation to an AR service and/or a specified profile.

If the wearable electronic device 301 detects a specified interaction,the wearable electronic device may determine a display mode (e.g., afirst display mode or a second display mode) for displaying informationon content depending on a processing entity for processing content beingexecuted in relation to an AR service. According to various embodimentsof the disclosure, the display mode may include a first display mode(e.g., including a mirroring-based display mode) of displaying contenton the AR screen through the display module 160 of the electronic device101 or a second display mode of mapping the content on the AR screen tothe electronic device 101 on the AR screen in a virtual space (e.g., anAR screen) through the display module 520 of the wearable electronicdevice 301 and displaying the content therein, depending on theprocessing entity of content

For example, in the case of the tethered method, the processing entityof content or an application of the content may be the electronic device101, and the first display mode may be determined as the display mode ofthe content. As another example, even in the tethered method, if thewearable electronic device 301 is able to perform processing (e.g.,switching of a processing entity) according to the properties ofcontent, or if a target device to display content is explicitlydesignated as the wearable electronic device 301 according to the user'sintention (e.g., the AR environment continues to be used), the seconddisplay mode may be determined as the display mode of the content.

In the case of the standalone method, the processing entity of contentor an application of the content may be the wearable electronic device301, and the second display mode may be determined as the display modeof the content. As another example, even in the standalone method, ifthe electronic device 101 is able to perform processing (e.g., switchingof a processing entity) according to the properties of content, or if atarget device to display content is explicitly designated as theelectronic device 101 according to the user's intention (e.g., mirroringor substantial extension of the display module 160 of the electronicdevice 101 in the real space), the first display mode may be determinedas the display mode for displaying the content.

The display mode determined according to the processing entity may varydepending on whether or not a profile of the content corresponds to aprofile specified in advance. For example, if the content corresponds toa specified profile, a display mode (e.g. the second display mode) usinga virtual space for the corresponding content (e.g., content of thespecified profile) may be determined, regardless of whether theprocessing entity is the electronic device 101 or the wearableelectronic device 301.

In operation 1711, based on the determination result, the wearableelectronic device 301 may determine whether the display mode is thefirst display mode or the second display mode. Based on the determineddisplay mode (e.g., the first display mode or the second display mode),the wearable electronic device 301 may operate to display the contentthrough the display module 160 of the electronic device 101 (e.g., thefirst display mode) or display the content on an AR screen through thedisplay module 520 of the wearable electronic device 301 (e.g., thesecond display mode).

If the display mode is determined to be the second display mode inoperation 1711, the wearable electronic device 301 may map the content,based on a specified area on the AR screen, in operation 1721. Forexample, the wearable electronic device 301 may determine an areacorresponding to the position and/or size determined for the content onthe AR screen, among a display area (e.g., a default area and/or anextension area extended according to sliding-out) associated with theelectronic device 101 and/or a virtual display area on the AR screen, asa specified area for displaying the content. The wearable electronicdevice 301 may identify a specified area (e.g., identify the defaultarea, the extension area, and/or the virtual area), configure an activewindow for the content, based on the specified area, and map the contentto the active window.

In operation 1723, the wearable electronic device 301 may provide (e.g.,display) the content through the active window on the AR screen.

In operation 1725, the wearable electronic device 301 may process a userinteraction (e.g., a user gesture input based on the content displayedon the AR screen) input onto the AR screen. The wearable electronicdevice 301 may track and detect a user interaction for controlling afunction related to the content on the AR screen. The wearableelectronic device 301 may track and detect a user interaction, based ontechnologies such as gaze tracking and object recognition, recognize apoint (e.g., a touch point) where a user interaction is detected in thecontent of the AR screen (e.g., recognize interaction inputinformation), perform processing to execute a related function, based onthe recognized result (e.g., touch input coordinates), and map thecontent according to the processing result to the AR screen (e.g., theactive window), thereby displaying the content.

The wearable electronic device 301 may perform user interaction for thecontent on the AR screen by the electronic device 101 even in the seconddisplay mode. For example, the user may perform user interaction throughthe electronic device 101 in the real space even in the second displaymode, and the electronic device 101 may calculate a point (e.g., a touchpoint) where the user interaction is detected and transmit relatedinformation (or interaction input information) (e.g., coordinates and/orspatial information) to the wearable electronic device 301 throughspecified communication connecting the wearable electronic device 301and the electronic device 101. The wearable electronic device 301 mayoperate to receive related information from the electronic device 101,recognize a point (e.g., a touch point) where the user interaction isdetected in the content on the AR screen, based on the received relatedinformation, execute a related function, based on the recognized result,reflect the processing result to the content, and display a resultscreen thereof (e.g., a screen in which the content is mapped to theelectronic device 101 on the AR screen).

If the display mode is determined as the first display mode in operation1711, the wearable electronic device 301 may map the content, based onthe electronic device 101 of the AR screen in operation 1731. Forexample, the wearable electronic device 301 may identify the state ofthe electronic device 101 on the AR screen (e.g., position, whether itis extended, and/or whether it is virtually extended) or identify aspecified user's intention (e.g., execute mirroring) and, based on theidentification result, determine information (e.g., a position and/orsize) about the display area for displaying the content in theelectronic device 101 of the AR screen. The wearable electronic device301 may map the content, based on the determined information on thedisplay area.

If the display mode is determined as the first display mode, thewearable electronic device 301 may operate to provide informationindicating the operation in the first display mode to the AR screen. Forexample, based on the determination of the first display mode, thewearable electronic device 301 may display, on the AR screen, a guidestating that the content is to be displayed through the electronicdevice 101 due to switching to the first display mode so that thewearable electronic device 301 is to be taken off, thereby providing theguide to the user.

Based on the determination of the first display mode, the wearableelectronic device 301 may stop displaying the screen of the wearableelectronic device 301 (e.g., turn off the display module 520) togetherwith the guide or subsequently thereto. For example, the wearableelectronic device 301 may operate to stop displaying the screen afterdisplaying the guide for a predetermined period of time. As anotherexample, the wearable electronic device 301 may operate to automaticallystop displaying the screen if it is taken off by the user. Whilestopping the screen display, the wearable electronic device 301 maycontinue to perform, as an internal (or background) operation,communication related to the processing of the content with theelectronic device 101 through specified communication connecting thewearable electronic device 301 and the electronic device 101. Forexample, the wearable electronic device 301 may an operation ofreceiving interaction input information from the electronic device 101through specified communication, processing content correspondingthereto, and transmitting processing results to the electronic device101.

In operation 1733, the wearable electronic device 301 may transmitmapping information to the electronic device 101. For example, thewearable electronic device 301 may produce mapping information relatedto content mapping and transmit the mapping information to theelectronic device 101 through specified communication.

In operation 1735, the electronic device 101 may identify a specifiedarea on the display module 160, based on the mapping information. Forexample, the electronic device 101 may receive mapping information fromthe wearable electronic device 301 and analyze the mapping informationto identify a specified area for displaying content in the displaymodule 160 of the electronic device 101. The electronic device 101 maydetermine a display area (e.g., a default area and/or an extension area)for displaying content in the display module 160 of the electronicdevice 101, based on the mapping information.

In operation 1737, the electronic device 101 may configure an activewindow for content, based on the specified area in the display module160.

In operation 1739, the electronic device 101 may provide (e.g., display)content through an active window in the display module 160.

In operation 1741, the electronic device 301 may process a userinteraction input through the electronic device 101 (e.g., a user'stouch input based on the displayed content). The electronic device 101may operate to detect a user interaction for controlling a functionrelated to the content displayed through the display module 160 andprocess a function corresponding to the user interaction.

The electronic device 101 may recognize a point (e.g., a touch point)where a user interaction is detected in the content and execute arelated function, based on the recognized result. The electronic device101 may display the content according to the processing result throughthe display module 160. The electronic device 101 may detect a userinteraction (e.g., a touch input) through the display module 160 andcalculate a point (e.g., a touch point) where the user interaction isdetected in the content provided through the display module 160. Theelectronic device 101 may transmit related information (or interactioninput information) (e.g., coordinates and/or spatial information) aboutthe point where the user interaction is detected to the wearableelectronic device 301.

The wearable electronic device 301 may recognize the point (e.g., atouch point) where the user interaction is detected in the content ofthe AR screen, based on the related information received from theelectronic device 101, and perform processing to execute a relatedfunction, based on the recognized result. The wearable electronic device301 may transmit content data according to the processing result to theelectronic device 101.

The electronic device 101 may receive the content data from the wearableelectronic device 301. The electronic device 101 may reflect thereceived content data to the content on the display module 160, therebydisplaying a result screen thereof (e.g., a screen on which content datais reflected).

FIGS. 18A to 18C are diagrams illustrating examples of operating adisplay mode between an electronic device and a wearable electronicdevice according to various embodiments of the disclosure.

FIGS. 18A, 18B, and 18C show examples of providing content through anelectronic device 101, based on determination of a first display mode,in an environment in which an electronic device 101 and a wearableelectronic device 301 provide an AR service, based on a tethered methodor a standalone method. FIGS. 18A, 18B, and 18C show examples in whichthe form factor of the electronic device 101 is a rollable type.

FIG. 18A shows an example in which the electronic device 101 enters anAR screen 1800 through the display module 520 of the wearable electronicdevice 301 and in which the user performs specified interaction (e.g., arollable activation interaction) for the electronic device 101 on the ARscreen 1800. For example, the user may move the electronic device 101into the AR screen 1800, and then input a specified gesture (e.g., aslide-out gesture or a rollable extension gesture) using the electronicdevice 101.

Referring to FIG. 18A, the specified gesture may include a gesture ofsubstantially extending the extension area of the electronic device 101(e.g., substantially sliding out the display module 160 of theelectronic device 101). The specified gesture may not be a substantialextension of the extension area of the electronic device 101. Forexample, the user may input only an extending gesture withoutsubstantially extending the electronic device 101. In an embodiment, thespecified gesture may be detected based on the user's hand detectionand/or hand tracking on the AR screen 1800.

FIGS. 18B and 18C show examples of providing content, which is beingoperated (e.g., displayed) through the AR screen 1800, through theactual display module 160 of the electronic device 101 in the real space(e.g., a first display mode), based on the display mode determined as afirst display mode.

FIG. 18B shows an example in which the display module 160 of theelectronic device 101 is extended in response to a specified interactionand in which the content 1810 being operated on the AR screen 1800 isdisplayed through an extension area B of the display module 160. Theelectronic device 101 may maintain the existing state of a default areaA of the display module 160 and provide the extension area B that isextended according to a specified interaction as an active window fordisplaying the content 1810.

FIG. 18C shows an example in which the display module 160 of theelectronic device 101 is extended in response to a specified interactionand in which the content 1820 being operated on the AR screen 1800 isdisplayed through the entire area including the default area A and theextension area B of the display module 160. For example, the specifiedinteraction by the user may reflect a clear intention of the user forextension of a screen. Accordingly, the electronic device 101 mayprovide the entire area (e.g., default area A+extension area B)including the default area A and the extension area B of the displaymodule 160 as an active window to display the content 1820.

An active window may be configured in at least one of the default areaA, the extension area B, or the entire area (e.g., default areaA+extension area B) of the electronic device 101, based on theoverlapping state (or overlapping area) of an affordance according tothe user's intention and the content. If the content 1810 and 1820overlaps the affordance, an active window may be configured based on theextension area B such that the corresponding content is to be displayedin the extension area B of the electronic device 101. If the content1810 and 1820 overlaps both the electronic device 101 (e.g., the defaultarea A of the electronic device 101) and the affordance on the ARscreen, an active window may be configured based on the entire area(e.g., default area A+extension area B) of the electronic device 101such that the corresponding content is to be displayed in the entirearea. If one piece of content 1810 or 1820 is disposed in theaffordance, the active window may be configured based on the entire area(e.g., default area A+extension area B) of the electronic device 101such that the corresponding content is to be displayed in the entirearea.

If a plurality of pieces of content (e.g., first content and secondcontent) overlaps the affordance, an active window may be configuredbased on each of the default area A and the extension area B such thatthe plurality of pieces of content is to be separately displayed in thedefault area A and the extension area B of the electronic device 101.One piece of content (e.g., first content) among a plurality of piecesof content overlaps the affordance and if another piece of content(e.g., second content) thereof overlaps the electronic device 101 (e.g.,the default area A of the electronic device 101), for example, if eachpiece of content is disposed according to the user's intention, a firstactive window for the first content may be configured in the extensionarea B and a second active window for the second content may beconfigured in the default area A so as to display the first content andthe second content to conform to the disposition according to the user'sintention.

The affordance may be varied (e.g., extended) by a user input (e.g., anaffordance extension interaction). If one piece of content (e.g., firstcontent) is disposed in (e.g., overlaps) the extended affordance, thedisplay module 160 of the electronic device 101 may be extended tocorrespond to the extended affordance and an active window may beconfigured so as to display one piece of content on the entire extendedscreen through the entire area (e.g., default area A+extension area B)including the extended extension area B and the default area A. If aplurality of pieces of content (e.g., first content and second content)is disposed in (e.g., overlaps) the extended affordance, an activewindow may be configured in each of the default area A and the extensionarea B so as to display the plurality of pieces of content in thedefault area A and the extension area B, respectively, or at least oneactive window may be configured in the extension area so as to displaythe plurality of pieces of content in the extension area B correspondingto the extended affordance.

If the electronic device 101 and/or the wearable electronic device 301determines to display content, based on the first display mode, duringoperation for the AR service, it may operate to stop displaying the ARscreen 1800 of the wearable electronic device 301, configure a specifiedactive window for the content being displayed on the AR screen throughthe default area and/or the extension area of the display module 160 ofthe electronic device 101, and display content based on the configuredactive window.

FIGS. 19A and 19B are diagrams illustrating an example of operating adisplay mode between an electronic device and a wearable electronicdevice according to various embodiments of the disclosure.

FIGS. 19A and 19B show examples of providing content through a wearableelectronic device 301, based on determination of a second display mode,in an environment in which an electronic device 101 and a wearableelectronic device 301 provide an AR service, based on a tethered methodor a standalone method. FIGS. 19A and 19B show examples in which theform factor of the electronic device 101 is a rollable type.

FIG. 19A, shows an example of mapping (e.g., texture-mapping) content1910 to the extension area of the electronic device 101 on the AR screen1900 and displaying the same. Referring to FIG. 19B, FIG. 19B shows astate of the electronic device 101 viewed in the real space, which may,as shown, show an example of the state in which content is not displayedin the extension area of the electronic device 101 in the real space.

Referring to FIGS. 19A and 19B, if the electronic device 101 and/or thewearable electronic device 301 determines to display content, based onthe second display mode, during operation for an AR service, content1910 may be displayed in association with the electronic device 101 onlythrough the AR screen 1900 of the wearable electronic device 301, forexample, through the extension area of the display module 160 of theelectronic device 101 on the AR screen 1900. Unlike this, the electronicdevice 101 in the real space may provide a corresponding area 1920(e.g., the extension area) as an empty space or inactive space (or to beturned off), instead of displaying the content 1910 displayed throughthe AR screen 1900.

As shown in FIGS. 19A and 19B, a first user wearing the wearableelectronic device 301 may check and control the content 1910 on the ARscreen 1900, whereas at least one second user in the real space isunable to check the content 1910 through the electronic device 101.

When providing the content 1910 to the AR screen 1900 based on thesecond display mode, the electronic device 101 and/or the wearableelectronic device 301 may further provide graphic processing for an areato which the content 1910 is mapped in the display module 160 of theelectronic device 101 on the AR screen 1900. For example, in order tosecure the user's visibility, the color (e.g., background color) of thearea to which the content 1910 is mapped in the display module 160 ofthe electronic device 101 on the AR screen 1900 may be virtuallychanged. It is possible to provide a graphic effect of providing thebackground of the area where the content 1910 is mapped and displayed asa color contrasting to a representative color of the content 1910 (e.g.,a dominant color).

FIGS. 20A to 20C are diagrams illustrating an example of operatingcontent mirroring and interaction between an electronic device and awearable electronic device according to various embodiments of thedisclosure.

FIGS. 20A, 20B, and 20C show examples of performing mirroring of atleast one piece of content on an AR screen to an electronic device 101through a wearable electronic device 301 and controlling at least onepiece of content on the AR screen, based on an interaction input throughthe electronic device 101, in an environment in which the electronicdevice 101 and the wearable electronic device 301 provide an AR service,based on a tethered method or a standalone method.

Referring to FIGS. 20A to 20C, the electronic device 101 and/or thewearable electronic device 301 may execute a mirroring function based ona user's intention while displaying content through the AR screenaccording to the second display mode. For example, the user may be inthe state of using an AR service by an application that is operable onlyin the wearable electronic device 301. The user may activate a mirroringfunction to display the content on the AR screen through the displaymodule 160 of the electronic device 101. In an embodiment, the mirroringfunction may include, for example, a function of mirroring content(e.g., mirror content) according to an application operable in thewearable electronic device 301 on the electronic device 101 whiledisplaying the content through the AR screen, thereby displaying thecontent through the display module 160 of the electronic device 101 aswell.

The wearable electronic device 301 may mirror a variety of controlinformation (e.g., components, objects, and/or windows) that the usermay check and control through the wearable electronic device 301 on theelectronic device 101 to be displayed thereon.

The mirroring function may enable the wearable electronic device 301 toperform processing on the content and the electronic device 101 todisplay the content transmitted (or mirrored) from the wearableelectronic device 301.

Referring to FIGS. 20A to 20C, the electronic device 101 may displaycontent 2010 transmitted from the wearable electronic device 301 throughthe display module 160. The user may use the electronic device 101 tocontrol content displayed on the AR screen of the wearable electronicdevice 301. For example, the user may input various gestures (e.g.,touch gestures) related to control of the content on the AR screen,based on the content 2010 displayed on the display module 160 of theelectronic device 101.

The user may perform, with respect to the content 2010 displayed on thedisplay module 160 of the electronic device 101, a first gesture input(e.g., FIG. 20A) for rotating the content 2010 by a screen touchoperation, a second gesture input (e.g., FIG. 20B) for enlarging thecontent 2010 by a screen zoom-in operation (e.g., pinch zoom-in), or athird gesture input (e.g., FIG. 20C) for reducing the content 2010 by ascreen zoom-out operation (e.g., pinch zoom-out).

Various gestures related to content control in the disclosure are notlimited thereto. The various gestures may include a gesture of movingthe content 2010 up, down, left, and right by a screen move operation ora gesture of moving the content 2010 back and forth by a screenmulti-touch move operation may be included.

The electronic device 101 may detect a user input shown in FIGS. 20A,20B, and/or 20C, calculate a point (e.g., a touch point) where the userinput is detected in the content on the display module 160, and transmitrelated information (e.g., coordinates and/or spatial information) tothe electronic device 101. The wearable electronic device 301 mayconvert the related information received from the electronic device 101into coordinate information on the AR screen and process the convertedcoordinate information as a user input for controlling the content ofthe AR screen.

The wearable electronic device 301 may perform processing of controllingthe content of the AR screen (e.g., execute a related function) tocorrespond to the user input based on the electronic device 101 andreflect the same on the content on the AR screen, thereby displaying aresult screen thereof. For example, the wearable electronic device 101may recognize a user input shown in FIGS. 20A, 20B, and/or 20C and, inresponse thereto, rotate, enlarge, or reduce the content on the ARscreen, thereby displaying a result screen thereof

FIGS. 21A to 21C are diagrams illustrating an example of an operation ofsharing content between an electronic device and a wearable electronicdevice according to various embodiments of the disclosure.

FIGS. 21A, 21B, and 21C show examples of providing content through anwearable electronic device 301, based on determination of a seconddisplay mode, and sharing at least a portion of content provided throughthe wearable electronic device 301 with an electronic device 101 in thereal space in an environment in which the electronic device 101 and thewearable electronic device 301 provide an AR service, based on atethered method or a standalone method. According to an embodiment,FIGS. 21A, 21B, and 21C may show an example in which the form factor ofthe electronic device 101 is a rollable type.

Referring to FIG. 21A, shows an example of mapping (e.g.,texture-mapping) content 2110 to an extension area of the electronicdevice 101 on an AR screen 2100 and displaying the content. Referring toFIG. 21B, FIG. 21B shows a state of the electronic device 101 viewed inthe real space, which may, as shown, show an example of the state inwhich content is not displayed in the extension area of the electronicdevice 101 in the real space.

Referring to FIGS. 21A and 21B, if the electronic device 101 and/or thewearable electronic device 301 determines to display content, based onthe second display mode, during operation for an AR service, content2110 may be displayed in association with the electronic device 101 onlythrough the AR screen 2100 of the wearable electronic device 301, forexample, through the extension area of the display module 160 of theelectronic device 101 on the AR screen 2100. Unlike this, the electronicdevice 101 in the real space may provide a corresponding area 2120(e.g., the extension area) as an empty space or inactive space (or to beturned off), instead of displaying the content 2110 displayed throughthe AR screen 2100.

Referring to FIGS. 21A and 21B, a first user wearing the wearableelectronic device 301 may check and control the content 2110 on the ARscreen 2100, whereas at least one second user in the real space isunable to check the content 2110 through the electronic device 101.

In the environment referring to FIGS. 21A and 21B, the electronic device101 and/or the wearable electronic device 301 may execute a sharingfunction based on the user's intention while displaying the content 2110through the AR screen 2110. The user may activate a sharing functionthat enables the content 2110 (or at least a portion (e.g., contentdata) of the content 2110) on the AR screen 2100 to be shared with theelectronic device 101 (e.g., transmission of content data of the content2110 (or file transmission) or partial disclosure thereof through thedisplay module 160 of the electronic device 101). In an embodiment, thesharing function may include, for example, a function in which thewearable electronic device 301 transmits the content 2110 (or at least aportion of the content 2110) to the electronic device 101 whiledisplaying the content 2110 on the AR screen 2100 to be shared with theelectronic device 101.

Referring to FIG. 21A, the electronic device 101 and/or the wearableelectronic device 301 may process a user interaction for the content2110 mapped to the electronic device 101 on the AR screen 2100 (e.g., auser's gesture input based on the content displayed on the AR screen2100). The electronic device 101 and/or the wearable electronic device301 may operate to track and detect a user interaction for executing asharing function to share the content 2110 (e.g., at least a portion ofthe content 2110) on AR screen 2100 and process a function of sharingthe content 2110 with the electronic device 101 in response to the userinteraction.

The user may input a gesture of selecting a specific object (e.g., textand/or an image) from the content 2110 mapped to a second area (e.g.,the extension area) of the electronic device 101 on the AR screen 2100and dragging the same to a first area (e.g., the default area) (e.g., agesture of dragging from the right area to the left area with referenceto the example in FIG. 21A). The wearable electronic device 301 maytrack and detect a user input related to a gaze sharing function andtransmit the content 2110 (e.g., a selected specific object) at a point(e.g., a touch point) where the user input is detected on the AR screen2100 to the electronic device 101 through specified communication.

The electronic device 101 may operate to detect reception of the content2110 (e.g., at least a portion of the content 2110) from the wearableelectronic device 301, store the received content 2110 in the memory 130of the electronic device 101, and/or display the same through thedisplay module 160. If the content 2110 is received from the wearableelectronic device 301, as shown in FIG. 21C, the electronic device 101may provide information from the wearable electronic device 301, statingthat the content 2110 is to be shared, to the user in a specified form(e.g., in a pop-up form). For example, the electronic device 101 mayprovide a guide window 2150 in the form of a pop-up, which informs thatthere is content 2110 to be shared with the wearable electronic device301 and enables receiving and storing the shared content 2110 orrejecting (cancelling) the reception of the shared content 2110.

When providing the guide window 2150 related to the content 2110 sharedby the user, as shown in FIG. 21C, the electronic device 101 may providethe guide window 2150 as a pop-up while maintaining the state shown inFIG. 21B (e.g., the inactive state of the extension area). If receptionand storage of the shared content 2110 is selected by the user, theelectronic device 101 may receive the shared content 2110 from thewearable electronic device 301 and store the same in the memory 130. Theelectronic device 101 may provide (e.g., display) the content 2110,which is received and stored, through a default area (e.g., the leftarea in FIG. 21A) of the display module 160. For example, the electronicdevice 101 may display the shared content 2110 through the default areawhile maintaining the inactive state of the extension area. Assuming astate in which the electronic device 101 displays a chat window throughthe default area, the electronic device 101 may operate to reflect theshared content 2110 from the wearable electronic device 301 on thechatting window in the form of an attached file and display the samewithin the chat window of the default area.

A method of providing an augmented reality (AR) service according to anembodiment of the disclosure may include providing content through an ARscreen, detecting a specified external object through the AR screenwhile providing the content, determining a display mode for providingthe content, based on the detection of the specified external object,and, based on the determined display mode, performing control to displaythe content through a display module of the specified external object(e.g., the display module 160 of the electronic device 101) orperforming control to display the content through a virtual display areaassociated with the specified external object on the AR screen.

The detecting of the external object may include detecting a specifiedinteraction through the AR screen.

The determining of the display mode may include determining whether atarget device processing the content is an electronic device 101 or awearable electronic device 301, based on the detection of the specifiedinteraction, if the target device is determined to be the electronicdevice 101, determining a first display mode in which the content on theAR screen switches to be displayed through the electronic device 101,and if the target device is determined to be the wearable electronicdevice 301, determining a second display mode in which the content ismapped to the electronic device 101 and displayed on the AR screen.

The performing control to display the content may include, based on thedetermination of the first display mode, performing control to stopdisplaying a screen of the wearable electronic device 301 andcontrolling the display of the content being displayed through the ARscreen, based on a display area corresponding to the display module 160of the electronic device 101.

The performing control to display the content may include configuring aspecified active window for the content being displayed on the AR screenthrough the default area and/or the extension area of the display module160 of the electronic device 101, and controlling display of contentbased on the configured active window.

The performing control to display the content may include, based on thedetermination of the second display mode, controlling display of contentbased on a virtual display area associated with the electronic device101 on an AR screen provided through the display module 590 of thewearable electronic device 301.

The performing control to display the content may include, based on thedetermination of the second display mode, configuring a virtual areacorresponding to the extension area of the display module 160 of theelectronic device 101 to correspond to the extension area of theelectronic device 101, configuring a specified active window for thecontent being displayed on the AR screen, based at least on the defaultarea, the extension area, and/or the virtual area of the display module160 of the electronic device 101, and controlling display of the contentthrough content mapping based on the configured active window.

If the content is provided according to the second display mode, thecontent mapped to the AR screen may be provided only through a displayarea associated with the electronic device 101 on the AR screen, insteadof being displayed through the display module 160 of the electronicdevice 101 in the real space.

The method may further indicate determining whether or not the contentcorresponds to a specified profile when determining the target deviceprocessing the content and, if the content corresponds to the specifiedprofile, determining the second display mode so as to provide content ofthe specified profile through the AR screen, regardless of the targetdevice processing the content.

While the disclosure has been shown and described with reference tovarious embodiments thereof, it will be understood by those skilled inthe art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the disclosure as definedby the appended claims and their equivalents.

What is claimed is:
 1. An augmented reality (AR) providing device for AR services, the AR providing device comprising: a display; and a processor, wherein the processor is configured to: provide content through an AR screen, detect a specified external object through the AR screen while providing the content, determine a display mode for providing the content, based on the detection of the specified external object, and based on the determined display mode, perform control to display the content through a display of the specified external object or perform control to display the content through a virtual display area associated with the specified external object on the AR screen.
 2. The device of claim 1, wherein the content is produced by the AR providing device for AR services and provided through the display of the AR providing device, or wherein the content is produced by the AR providing device, transmitted to an external device connected through specified communication, and provided through a display of the external device.
 3. The device of claim 1, wherein the processor is further configured to: detect a specified interaction through the AR screen, determine whether a target device processing the content is an electronic device or a wearable electronic device, based on the detection of the specified interaction, in case that the target device is determined to be the electronic device, determine a first display mode in which the content on the AR screen switches to be displayed through the electronic device, and in case that the target device is determined to be the wearable electronic device, determine a second display mode in which the content is mapped to the electronic device and displayed on the AR screen.
 4. The device of claim 3, wherein the processor is further configured to: perform control to stop displaying a screen of the wearable electronic device, based on the determination of the first display mode, and control the display of the content being displayed through the AR screen, based on a display area corresponding to a display of the electronic device.
 5. The device of claim 4, wherein the processor is further configured to: configure a specified active window for the content being displayed on the AR screen through a default area and/or an extension area of the display of the electronic device, and control display of the content based on the configured active window.
 6. The device of claim 3, wherein the processor is further configured to, based on the determination of the second display mode, control display of the content based on a virtual display area associated with the electronic device on an AR screen provided through a display of the wearable electronic device.
 7. The device of claim 6, wherein the processor is further configured to, based on the determination of the second display mode, map and display the content, based at least on a default area of the electronic device positioned on an AR screen provided through the display of the wearable electronic device, an extension area of the electronic device, and/or a virtual area produced adjacent to the electronic device.
 8. The device of claim 6, wherein the processor is further configured to: configure a virtual area corresponding to an extension area of the display of the electronic device to correspond to the extension area of the electronic device, based on the determination of the second display mode, configure a specified active window for the content being displayed on the AR screen through a default area and/or the extension area of the display of the electronic device, and control display of the content through content mapping based on the configured active window.
 9. The device of claim 6, wherein in case that the processor provides the content according to the second display mode, the content mapped to the AR screen is provided only through a display area associated with the electronic device on the AR screen, instead of being displayed through the display of the electronic device in a real space.
 10. The device of claim 3, wherein the specified interaction comprises: a first interaction in which the electronic device enters the AR screen provided through the display of the wearable electronic device, a second interaction in which a user inputs a specified gesture using the electronic device after the electronic device enters the AR screen, and/or a third interaction in which the user inputs a specified menu manipulation gesture through the electronic device on the AR screen after the electronic device enters the AR screen.
 11. The device of claim 3, wherein the processor is further configured to: determine whether or not the content corresponds to a specified profile in case that determining the target device processing the content, and in case that the content corresponds to the specified profile, determine the second display mode so as to provide content of the specified profile through the AR screen, regardless of the target device processing the content.
 12. The device of claim 1, wherein the processor is further configured to: recognize an external device corresponding to the external object on the AR screen while displaying the content through the display, monitor a change in state of the external device in case that the external device is a specified external device, determine a processing entity related to the content, based on identifying a change in the state of the external device, determine the display mode of the content, based on the determined processing entity, and based on the determined display mode, display the content on the AR screen through a display or control the external device to display the content through a display of the external device.
 13. A method of providing an augmented reality (AR) service, the method comprising: providing content through an AR screen; detecting a specified external object through the AR screen while providing the content; determining a display mode for providing the content, based on the detection of the specified external object; and based on the determined display mode, performing control to display the content through a display of the specified external object or performing control to display the content through a virtual display area associated with the specified external object on the AR screen.
 14. The method of claim 13, wherein the detecting of the external object comprises detecting a specified interaction through the AR screen, and wherein the determining of the display mode comprises: determining whether a target device processing the content is an electronic device or a wearable electronic device, based on the detection of the specified interaction, in case that the target device is determined to be the electronic device, determining a first display mode in which the content on the AR screen switches to be displayed through the electronic device, and in case that the target device is determined to be the wearable electronic device, determining a second display mode in which the content is mapped to the electronic device and displayed on the AR screen.
 15. The method of claim 14, wherein the performing control to display the content comprises: based on the determination of the first display mode, performing control to stop displaying a screen of the wearable electronic device and controlling the display of the content being displayed through the AR screen, based on a display area corresponding to a display of the electronic device, and based on the determination of the second display mode, controlling display of content based on a virtual display area associated with the electronic device on an AR screen provided through a display of the wearable electronic device.
 16. The method of claim 15, wherein the performing control to display the content comprises: configuring a virtual area corresponding to an extension area of the display of the electronic device to correspond to the extension area of the electronic device, based on the determination of the second display mode, configuring a specified active window for the content being displayed on the AR screen through a default area and/or the extension area of the display of the electronic device, and controlling display of the content through content mapping based on the configured active window.
 17. The method of claim 14, further comprising, while operating in the second display mode, continuing to display a screen displayed on the electronic device.
 18. The method of claim 14, further comprising, while operating in the second display mode, displaying a blank area in an area of a display of the electronic device corresponding to an extension area.
 19. The method of claim 14, further comprising, while operating in the second display mode, executing a sharing function enabling content displayed on the AR screen to be shared with the electronic device.
 20. The method of claim 19, wherein the sharing function is a function in which at least a portion of content displayed on the AR screen is transmitted to the electronic device. 